# AI-Based Reporting
Source: https://knowledge.cloudquant.com/administration/ai-based-reporting
Ask natural-language questions about Liberator usage, users, and entitlements from the Admin UI
# AI-based reporting
The **AI Reporting** panel is a natural-language assistant in the Liberator Admin UI. Admins can ask about platform usage, users, permissions, and entitlements in plain English, or request a downloadable report, without building manual queries or filters.
Available in Liberator **2.3** and later. The panel is labeled **BETA**.
## Who can use it
AI-based reporting is an **Admin / Super Admin** capability in the Liberator Admin UI. Results respect the same platform data the admin portal already uses for usage and entitlement views.
## Quick-action reports
The panel includes preset buttons that run canned reports and return a narrative summary (and an export when applicable):
| Quick action | What it returns |
| ----------------------------- | ----------------------------------------------- |
| **Usage summary (yesterday)** | High-level usage narrative for the previous day |
| **Usage report (yesterday)** | Detailed usage report for the previous day |
| **User report (today)** | User activity report for the current day |
| **Current entitlements** | Snapshot of current entitlement grants |
| **Usage report (last month)** | Usage for the previous calendar month |
| **Usage report (this month)** | Usage for the current calendar month |
## Free-form questions
Type requests in natural language, for example:
* "Give me the usage for `` for the second week of February."
* "Which users were most active yesterday by query count?"
* "Show entitlements for ``."
The assistant resolves the request against live data: it identifies the relevant dataset(s), computes per-dataset or per-user totals (queries, records, bytes, GB), and can generate a downloadable Excel (`.xlsx`) export.
### Username clarification
If a typed username does not match a real account, the assistant asks you to confirm or correct it and suggests likely matches. After you confirm, it continues with the corrected user without requiring you to retype the full request.
### Empty periods
If no usage occurred in the requested window, the assistant still produces a valid (header-only) export and explains that there was no activity, rather than returning an error.
## Excel exports
Reports from quick actions or free-form questions can be downloaded as `.xlsx` files. The chat response includes the filename and file size. Typical columns include user, dataset, query count, record count, bytes, and gigabytes, matching the summary shown in the panel.
## Related
Per-query records (2.4+) — who queried which dataset, window, rows, and outcome
Provision users and grant dataset access
Liberator 2.3 release notes
# AI-Driven Dataset Onboarding
Source: https://knowledge.cloudquant.com/administration/ai-dataset-onboarding
Use the AI Assisted flow in the Create Dataset wizard to describe a source in plain language and auto-fill connection and dataset settings
# AI-driven dataset onboarding
AI Onboarding is an assisted setup flow built into Liberator's **Create Dataset** wizard. After you pick a source type, choose **AI Assisted** to describe your data source in plain language (or paste connection details). The assistant tests the connection and fills the standard wizard steps for you to review before creating.
Available in Liberator **2.3** and later. The flow is labeled **BETA**. Manual setup remains available for every connector.
## Who can use it
Admins and Super Admins who can create connections and datasets. AI Assisted is offered for every connector type, including S3, Snowflake, SFTP, SQL Server, MySQL, Oracle, PostgreSQL, OneDrive, SharePoint, Azure Blob, Google Cloud Storage, CIFS, FTPS, Local File, and HubSpot.
## Choose AI Assisted or Manual
1. Go to **Datasets → Add Dataset** (or start from **Connections**).
2. Select a source type.
3. Choose a setup flow:
| Flow | When to use it |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| **AI Assisted (BETA)** | Paste credentials and a short description; let Liberator fill connection, config, dataset details, and column metadata |
| **Manual** | Step through every connection and dataset field yourself |
## Describe your source
Choosing **AI Assisted** opens a chat panel with a prompt such as **Describe your \ Dataset**. You can:
* Paste raw connection details (host, keys, bucket, folder/prefix, table name, and so on)
* Describe the source in plain language
* Include wildcard or prefix file patterns (for example, `reports/mag_7_fake_*.csv`) — patterns are preserved exactly in the folder/prefix field
If you do not include a dataset name, the assistant asks for one before continuing. The dataset name becomes the queryable identifier.
## What the assistant fills in
On a successful connection test, the assistant populates the standard wizard steps:
| Wizard step | What AI fills |
| ------------------- | ------------------------------------------------------------------------------ |
| **Basic Info** | Dataset name (if provided or confirmed in chat) |
| **Connection** | Credentials, endpoint, bucket/account, and related connection fields |
| **Configuration** | Default timezone, detected date/time column(s) and format, sample-rows preview |
| **Dataset Details** | Suggested queryable key |
| **Column Metadata** | Column type, display name, and an AI-generated description for each column |
| **Webhooks** | Left for you to configure if needed |
Every AI-filled field stays editable. Nothing is locked until you click **Create Connection** / **Create Dataset** (or the equivalent continue action) at each step.
## Review before creating
The assistant asks you to review and edit anything that looks off. Typical checks:
* Connection test reported success
* Folder/prefix or table matches the intended source
* Date/time column and format look correct
* Sample rows show the expected data
* Column descriptions and the queryable key are usable as-is or edited
## In-flow help
You can ask clarifying questions about the current wizard (for example, what a translation column is) without restarting onboarding. The assistant stays scoped to connecting and configuring **this** dataset — off-topic or destructive requests (such as deleting other datasets) are declined and the conversation is redirected to the outstanding connection details.
## After creation
The new dataset appears in the **Datasets** list with the name you confirmed and the correct source-type badge. You can still edit connection or dataset settings later through the normal Liberator UI.
## Related
AI-assisted and frequency-ranked column descriptions
Connection + dataset architecture and source types
Natural-language usage and entitlement reports
Liberator 2.3 release notes
# Cache Pre-Generation
Source: https://knowledge.cloudquant.com/administration/cache-pre-generation
Schedule dataset cache creation ahead of time so users query warm Parquet caches instead of cold source systems
# Cache pre-generation
By default, Liberator builds its cache the first time a query runs (cache warming). For large datasets or slow source databases, that first query can take a long time. **Pre-generated cache** lets Super Admins schedule cache creation ahead of time so data is ready when users query it.
Cache files are written in **Parquet** format (replacing the older Arrow cache format) and can be stored in **Amazon S3** or **Google Cloud Storage**. Cached data does not need to live on the Liberator server. Parquet is typically much more compact than the previous Arrow caches, which reduces storage cost at scale.
Super Admin role is required to configure cache storage connections and pre-generation settings.
## Step 1 — Create a cache storage connection
Before enabling pre-generated cache on a dataset, create a cache storage destination.
Go to **Connections → Add Connection**.
Select **Cache Storage** as the connection type.
Select your storage provider (**S3** is supported today; confirm GCS availability with your CloudQuant account team if needed). Enter the bucket name and credentials.
Click **Test Connection**, then **Save**.
## Step 2 — Enable pre-generated cache on a dataset
Open the dataset and click **Edit**.
Scroll to **Advanced Options** and toggle on **Pre-Generated Cache**.
Select **Parquet** as the cache format (recommended). Under **Destination**, select the cache storage connection from Step 1.
Click **Test** next to the destination to confirm connectivity, then save.
## Step 3 — Configure the cache window
| Option | Use when |
| -------------------- | ---------------------------------------------------------------------------------------- |
| **Rolling window** | Users mostly query recent data; keeps the last N days cached and advances automatically. |
| **Fixed date range** | You need a known historical slice that does not change. |
| **Full dataset** | You want complete coverage and the dataset is bounded in size. |
Enter the number of days (rolling) or the start/end dates (fixed), then save.
Long or open-ended retention periods trigger a **storage impact warning** in the UI. Review projected volume before saving — large windows in Parquet can accumulate significant object storage over time.
## Step 4 — Run or schedule generation
**Scheduled:** Once a schedule is configured on the dataset, cache generation runs automatically. No further action is required.
**Manual:** Open the dataset and click **Trigger Cache Pre-Generation**. A status indicator shows whether the job is in progress or complete.
## What users experience
After pre-generated cache is populated, queries against that dataset are much faster because Liberator reads the warm cache instead of the source system. This matters most for large SQL-backed datasets where the first cold query previously took minutes.
## Tips
* Use **rolling window** for datasets queried over recent windows (for example, the last 30 days of market data).
* Use **fixed date range** for historical snapshots that do not change.
* Use **full dataset** only when storage budget and dataset size are understood.
* The **Most queried datasets** view on [System Monitoring](/system-monitoring/overview) helps prioritize which datasets to pre-generate.
# Column Description Suggestions
Source: https://knowledge.cloudquant.com/administration/column-description-suggestions
Reuse and AI-assisted column descriptions when onboarding or editing dataset metadata
# Column description suggestions
When you onboard a dataset or edit column metadata, Liberator can **suggest descriptions** already in use on similar columns in your environment, ranked by how often each description appears. Accept a suggestion as-is, edit it, or write your own.
In Liberator **2.3** and later, suggestions can also be **AI-assisted**: Liberator proposes human-readable descriptions for cryptic or undocumented columns, then ranks them alongside your existing description library.
During **AI Assisted** dataset creation, column descriptions are generated as part of the wizard fill. See [AI-driven dataset onboarding](/administration/ai-dataset-onboarding).
This keeps documentation consistent across related datasets and speeds up onboarding for common fields (timestamps, tickers, prices, identifiers).
Admin privileges or higher are required. Suggestions improve as more datasets with descriptions exist in your environment.
## Use suggested descriptions
Go to **Datasets → Add Dataset** or **Edit** on an existing dataset.
Proceed to the **Columns** step.
For each column, Liberator shows likely description matches ordered by frequency of use.
Click a suggestion to apply it, edit the text in place, or type a new description manually.
Complete the wizard and save. New and edited descriptions feed future suggestions.
## Tips
* Use **consistent terminology** (for example, always "Trade execution timestamp (UTC)") so rankings stay useful.
* **Edit** close matches instead of creating near-duplicates that compete in frequency ranking.
* Treat low-frequency suggestions cautiously — they may not generalize.
## FAQ
**No suggestion for a column?**\
The field is blank until you enter one manually. It will be available for future datasets with similarly named columns.
**Can I ignore suggestions?**\
Yes. Suggestions are advisory only.
**Do suggestions cross public and private datasets?**\
Yes. Descriptions are metadata drawn from all datasets in your environment, not from underlying row data.
# Dataset Field Mapping
Source: https://knowledge.cloudquant.com/administration/dataset-field-mapping
Map source dataset columns to reference lookup tables and enrich query results with translate=true
# Dataset field mapping
Dataset field mapping defines a relationship between a **source** dataset and one or more **reference** (lookup) datasets. When a user queries the source with translation enabled, Liberator enriches values from the reference data at query time. Source data is never modified.
**Example:** A transactions dataset stores country codes (`US`, `DE`, `JP`). A reference dataset maps codes to English country names. With mapping configured, `translate=true` returns the country name automatically.
You can attach **multiple lookup tables** to one source dataset (products, locations, employees, and so on) and resolve them in a single query.
Admin privileges or higher are required. The reference dataset must already exist in Liberator before you map it.
## Step 1 — Prepare the reference dataset
Onboard lookup data as a standard dataset (CSV, database table, etc.). Include at least:
* A **key** column to match on (for example, `country_code`)
* One or more **output** columns (for example, `english_name`)
## Step 2 — Add translation columns
Navigate to the source dataset users will query and click **Edit**.
Go to the **Columns** step and click **Add Translation Columns**.
## Step 3 — Configure the mapping
In the **Add Translation Columns** dialog:
1. **Select reference dataset** — choose the lookup dataset.
2. **Source column (join key)** — column in the source dataset to match (for example, `country_code`).
3. **Reference column (join key)** — corresponding column in the reference dataset (for example, `alpha_2_code`).
4. **Output column(s)** — reference columns to surface in results (for example, `english_name`).
Click **Add**. Repeat for additional lookup tables.
## Step 4 — Save and query
Save the dataset. When querying, pass the `translate` parameter:
| Parameter | Result |
| ----------------- | --------------------------------------------------------- |
| `translate=true` | Returns enriched/translated values from attached lookups. |
| `translate=false` | Returns raw source data only. |
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name="transactions",
symbols=["AAPL"],
back_to="2026-01-01",
as_of="2026-01-31",
translate=True,
))
```
See [Query parameters](/api-reference/concepts/query-parameters) for `translate` in other clients.
## Conflict and missing-value behavior
| Situation | Behavior |
| --------------------------------- | --------------------------------------------------- |
| No match in the lookup table | Output column is `null`; the row is still returned. |
| Multiple entries for the same key | The entry with the **most recent timestamp** wins. |
| Multiple lookup tables | All lookups resolve in one query pass. |
## Maintenance
* Edit mappings from **Edit → Columns** on the source dataset.
* Changes take effect on the next query.
* Deleting a mapping entry causes affected output columns to return `null`.
## Tips
* Keep reference datasets small and focused on mapping data only.
* Split large flat mapping tables into per-topic lookup datasets when possible.
* Test with `translate=true` on a narrow date range before rolling out to users.
# Dataset Visibility
Source: https://knowledge.cloudquant.com/administration/dataset-visibility
Control whether datasets are public (discoverable by all authenticated users) or private (entitlement required)
# Dataset visibility
Every dataset has a **Publicly Available** toggle that controls whether authenticated users can discover and query it without an individual entitlement.
| Visibility | UI toggle | Who can see and query |
| ----------- | -------------------------- | --------------------------------------------------------- |
| **Public** | **Publicly Available** ON | Any authenticated user — no per-user entitlement required |
| **Private** | **Publicly Available** OFF | Only users with an explicit entitlement |
Authentication is always required. Unauthenticated requests never return datasets.
Visibility and entitlements are separate. A **private** dataset still requires an entitlement assignment on the [Users](/administration/managing-users-and-entitlements) page. A **public** dataset is queryable by any signed-in user without that step.
## Who can change visibility
| Role | Can set public / private |
| --------------- | ------------------------ |
| **Super Admin** | Yes |
| **Admin** | Yes |
| **User** | No |
## Set visibility during onboarding
As Admin or Super Admin, go to **Datasets → Add Dataset**.
Proceed through source, connection, and columns.
Near the top of the configuration page, set **Publicly Available** ON (public) or OFF (private).
Complete the wizard and save. The setting takes effect immediately.
## Change visibility on an existing dataset
1. Go to **Datasets** in the admin portal.
2. Use the **Public** column in the list to audit current visibility at a glance.
3. Open the dataset, toggle **Publicly Available**, and save.
Making a public dataset private hides it from non-entitled users immediately.
## API and catalog behavior
* **Private** datasets do not appear in catalog browse or search for users without entitlement.
* **Public** datasets appear in the catalog for any authenticated user.
* Use `liberator.datasets(entitled=True)` to list datasets you can access (entitlements plus public datasets you are allowed to see). See [Checking dataset access](/python-guide/dataset-access) and [Listing datasets](/api-reference/concepts/listing-datasets).
## Pre-generated cache
Visibility does not change cache behavior. Pre-generated cache continues to serve entitled users if you switch a dataset from public to private.
## Recommended practices
* **Public** — shared reference data (calendars, instrument masters) that all users need.
* **Private** — client-specific, team-scoped, or sensitive datasets.
* Audit the **Public** column periodically; test datasets left public during onboarding are a common oversight.
## Troubleshooting
**User cannot see an expected dataset** — If private, confirm entitlement on the Users page. If public, confirm the user can authenticate.
**Dataset visible to users who should not see it** — Set **Publicly Available** OFF and save.
**Toggle not visible** — Confirm you are signed in as Admin or Super Admin.
## Platform datasets
Some datasets are private by design and are not managed with **Publicly Available**. In Liberator **2.4** and later, **`usage_log`** is Super Admin only and is hidden from the standard user catalog. See [Query usage log](/administration/usage-log).
# File Upload Management
Source: https://knowledge.cloudquant.com/administration/file-upload-management
Upload, replace, download, and delete files behind file-upload connections without backend access
# File upload management
The **File Management** panel on a file-upload connection lets Admins manage the physical file behind a dataset: upload, replace, download, or delete — without access to underlying storage.
Supported formats for upload connections include `.csv`, `.tsv`, `.psv`, `.txt`, and `.xlsx`. See [Supported Data Formats](/datasource-config/supported-formats) for the full list.
Admin privileges or higher on the connection are required. The panel appears only on **File Upload** connections, not S3, SharePoint, database, or API sources.
## Open File Management
1. Sign in to the admin portal.
2. Go to **Connections** and open the connection.
3. Scroll to **File Management**.
| Panel state | Meaning | Action |
| ---------------------- | ----------------------- | ------------------------------------- |
| Empty (no file listed) | No file associated yet | **Upload** |
| File name shown | File already associated | **Replace**, **Download**, **Delete** |
## Upload a new file
Use when the connection has no file yet, or after a delete.
The panel shows no file name and an **Upload** control.
Click **Upload**, select the file, and wait for the progress indicator to complete.
Confirm the file name appears and run a small test query on the linked dataset.
If the dataset was newly created, finish column, date, and key configuration before notifying users.
## Replace an existing file
Use when data was refreshed or corrected but dataset name, columns, entitlements, and mappings should stay the same.
The replacement file must use the **same format** and **same column structure** as the original. Added, removed, or renamed columns require re-onboarding instead of replace.
Use **Download** to inspect the current file first.
Click **Replace**, select the new file, and wait for completion.
Query a value you know exists only in the new file to confirm the swap.
If **pre-generated cache** is enabled, trigger **Trigger Cache Pre-Generation** or wait for the next scheduled run so queries do not serve stale cache from the old file.
## After upload or replace
| Aspect | Behavior |
| ----------------------- | ------------------------------------------ |
| Dataset name | Unchanged |
| Column configuration | Unchanged — verify headers still align |
| Entitlements | Unchanged |
| Translation mappings | Resolve against the new file on next query |
| Previous file (replace) | Permanently removed |
## Troubleshooting
**Upload/Replace not visible** — Confirm Admin+ role and File Upload connection type.
**Validation error** — Check delimiter, encoding, and that format matches (`.csv` for `.csv`).
**No data after upload** — Column headers or date/key fields may not match the file; re-onboard if structure changed.
**Stale data after replace** — Refresh pre-generated cache if enabled.
# Fine-Grained Permissions
Source: https://knowledge.cloudquant.com/administration/fine-grained-permissions
Restrict user query scope with trial periods, date ranges, row limits, keys, and column filters
# Fine-grained permissions
Fine-grained permissions let Super Admins apply additional restrictions on top of standard dataset entitlements. Use them to enforce trial periods, limit historical lookback, cap query volume, or restrict which keys and columns a user can access.
Fine-grained rules apply **per dataset** or **per dataset group** and are configured from the user's **Data Permissions** dialog.
## When to use fine-grained permissions
| Scenario | Recommended restriction |
| ----------------------------------- | --------------------------------------- |
| Time-limited trial | **Trial Range** |
| Historical data only (no live feed) | **Data Query Range** → Historical Only |
| Rolling window (e.g., last 90 days) | **Data Query Range** → Rolling Window |
| Fixed evaluation period | **Data Query Range** → Fixed Date Range |
| Cap the width of a time-range query | **Max Query Days** |
| Cap download size | **Max Rows** |
| Limit API usage | **Max Queries** |
| Restrict to a symbol universe | **Keys** |
| Hide sensitive columns | **Columns** |
Fine-grained permissions require an underlying dataset or group entitlement. They cannot grant access on their own — the user must already be entitled to the dataset or group.
## Opening fine-grained settings
From the Users page, click the data-permissions icon for the target user.
Ensure the dataset or dataset group row is checked in the permissions table.
Click the gear icon (**Actions** column) on the selected row. The icon appears filled (blue) when fine-grained rules already exist for that item.
Enable the restriction sections you need, set values, click **Done** to return to the permissions table, then **Save Changes**.
Fine-grained settings are not saved until you click **Save Changes** on the main Data Permissions dialog. Clicking **Done** only returns you to the entitlement table.
## Restriction sections
Each section is controlled by an **Enable** toggle. Disabled sections are ignored — their fields are not sent to the backend.
### Trial Range
Limits when the user can access the platform for a specific dataset or group, independent of the data date range.
| Field | Description |
| ---------------- | ---------------------------------------- |
| **Trial Period** | Start and end dates for the trial window |
Outside the trial window, queries against the restricted dataset or group are denied even if other entitlements exist.
### Data Query Range
Controls what type of data the user can query and over what time horizon.
| Type | Behavior |
| -------------------------- | -------------------------------------------------------------------------------- |
| **Historical Only** | Live/real-time data is blocked; only historical queries are allowed |
| **Live Only** | Only live/real-time data is accessible |
| **Both Historical & Live** | No restriction on query mode |
| **Fixed Date Range** | Queries are limited to a specific start/end date range |
| **Rolling Window** | Queries are limited to the last N days from the current date (**Days Lookback**) |
### Limits and Settings
Caps query scope and restricts the symbol universe and visible columns.
| Field | Description |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Max Rows** | Maximum number of rows returned per query |
| **Max Query Days** (2.4+) | Maximum number of calendar days a single query's `back_to`–`as_of` window may span. Use this on high-volume datasets so one oversized request cannot stall other users. |
| **Max Queries** | Maximum number of queries allowed (usage cap) |
| **Max Keys** | Maximum number of keys (symbols) that can be queried in a single request |
| **Keys** | Allow-list of specific keys. Enter individually or paste comma-separated values |
| **Columns** | Allow-list of column names from the dataset schema. Search and multi-select from available columns |
System columns (`_seq`, `timestamp`, `muts`, `symbol`) are excluded from the column picker because they are always required for query execution.
## Dataset vs group rules
Fine-grained permissions can target either:
* **A single dataset** — restrictions apply only to that dataset
* **A dataset group** — restrictions apply to the group entitlement as a whole
Group-level rules are configured the same way: select the group row, click the gear icon, and set restrictions. The dialog title shows the group name.
## Removing fine-grained permissions
To remove all restrictions for a dataset or group:
1. Open the fine-grained settings (gear icon).
2. Disable all three sections (**Trial Range**, **Data Query Range**, **Limits and Settings**), or clear all restriction values.
3. Click **Done**, then **Save Changes**.
Alternatively, revoke the entire dataset or group entitlement — removing an entitlement automatically removes its associated fine-grained rules.
## Interaction with connection entitlements
Datasets inherited from a **connection-level** entitlement cannot have individual fine-grained rules configured in the per-user permissions UI. The gear icon is hidden for locked (inherited) datasets.
To apply fine-grained restrictions in this case, either:
* Assign a direct dataset entitlement instead of (or in addition to) the connection entitlement, or
* Manage access at the connection level through connection permission settings
## How restrictions are enforced
When a user submits a query, the entitlements service evaluates:
1. Whether the user has a dataset, group, or connection entitlement
2. Whether any fine-grained rules apply to that dataset
3. Whether the query parameters (date range, day span, keys, columns, row count) fall within the allowed bounds
If a query violates a restriction, it is rejected before data is returned. In Liberator **2.4** and later, the error names the limit that was exceeded (for example max rows, max query days, or missing entitlement) and the configured value. Callers can correct the request instead of treating every rejection as "not entitled."
## Example configurations
### 90-day rolling trial
1. Enable **Trial Range** and set the trial period dates.
2. Enable **Data Query Range**, select **Rolling Window**, set **Days Lookback** to `90`.
3. Save.
### Cap a single query's date span (2.4+)
1. Enable **Limits and Settings**.
2. Set **Max Query Days** to the widest window you will allow (for example `7` or `30`).
3. Optionally set **Max Rows** as a second backstop.
4. Save.
### Symbol-restricted research access
1. Enable **Limits and Settings**.
2. Set **Keys** to the allowed symbol list (e.g., `AAPL`, `MSFT`, `GOOG`).
3. Optionally set **Max Rows** to cap result size.
4. Save.
### Column-redacted dataset
1. Enable **Limits and Settings**.
2. Use **Columns** to select only the fields the user should see.
3. Save.
## Checking effective access
Users can verify entitled datasets from Python:
```python theme={null}
liberator.datasets(entitled=True)
```
Fine-grained restrictions are enforced at query time and are not visible in the dataset list. If a query fails due to a restriction, the error message indicates which limit was exceeded and, in Liberator **2.4** and later, the configured cap.
For admin-side verification, open the user's **View** dialog or the **JSON** tab (Super Admin) to inspect the full permissions payload including `fine_grain_permissions`.
# Managing Users & Entitlements
Source: https://knowledge.cloudquant.com/administration/managing-users-and-entitlements
Add, create, deactivate users and grant or revoke dataset entitlements in the CloudQuant Data Liberator admin portal
# Managing users and entitlements
This guide covers the full user lifecycle in the CloudQuant Data Liberator admin portal — from onboarding a new account to assigning and revoking dataset access.
## Prerequisites
* **Super Admin** role for creating users, editing details, assigning entitlements, and deactivating users
* **Admin** or **Super Admin** role to add existing user accounts to the permissions system
Navigate to **Users** in the admin portal sidebar.
## User lifecycle
```mermaid theme={null}
flowchart LR
A[Create or provision account] --> B[Add to Liberator]
B --> C[Assign entitlements]
C --> D[Optional fine-grained rules]
D --> E[User queries data]
E --> F[Deactivate when offboarded]
F --> G[Re-activate if needed]
```
## Creating a new user
Super Admins can create a brand-new account directly from the Users page.
Click **Create User** in the top-right of the Users page.
Fill in the required fields:
* **Username** — unique login identifier
* **Email**
* **First name** and **Last name**
* **Company** — select an existing company or type a new one
* **Password** — minimum 8 characters; use the generate button for a secure password (not required when using invite-via-email)
Click **Create**. The platform account is created.
In Liberator **2.4** and later, create and enroll happen in **one step**. The new user is added to Liberator permissions as part of Create User, so you do not need a separate **Add Users** pass for accounts you just created. Use **Add Users** only for existing platform accounts that are not yet in Liberator.
## Invite via email
In Liberator **2.3** and later, Super Admins can provision users without sharing temporary passwords out of band.
Open **Create User** and turn on **Invite via email**.
Provide username, email, name, and company as usual. Liberator sends the invite to the email address you enter.
The recipient opens the invite link, verifies their email, and sets their own password. Liberator requires a password change on first login for newly invited accounts.
In Liberator **2.4** and later, invite and email-verification messages include the username the recipient should sign in with.
### Send a password-reset email
From the Users table **Actions** menu, select **Send reset email** to trigger a self-service password reset for an existing account. Use this when a user cannot sign in and you do not want to set a temporary password yourself.
## Adding users to Liberator
Before a user can receive dataset entitlements, their account must exist in the Liberator permissions system. Use **Add Users** to enroll accounts that are not yet active in Liberator.
Click **Add Users** on the Users page.
The **Available Users** tab lists enabled accounts that are not yet in the permissions system. Use the column filters to search by name, email, or (in Liberator **2.4** and later) **company**, select one or more users, then click **Add**.
The **Deactivated Users** tab lists previously deactivated accounts. Select users and click **Re-activate** to restore their Liberator enrollment. You will still need to re-assign entitlements if they were removed during deactivation.
You can add multiple users in a single operation. Selected users appear as pills above the table for easy review before confirming.
## Editing user details
Super Admins can update profile information and platform roles from the Users table.
In the **Actions** column, click the edit icon for the target user.
Modify any of:
* First name, last name, email, company
* **User Role** — `User`, `Admin`, or `Super Admin` (you can only assign roles at or below your own level)
* **Password** — optionally set a temporary password (copied to clipboard on save)
Click **Save Changes**.
## Assigning dataset entitlements
Entitlements control which datasets a user can query. Super Admins manage entitlements through the **Data Permissions** dialog.
In the Users table **Actions** column, click the data-permissions icon for the target user.
The permissions table lists all datasets and dataset groups. Check the rows you want to grant:
* **Dataset** rows grant access to a single dataset
* **Group** rows grant access to every dataset in that dataset group
Selected items appear as pills above the table. Remove a pill to revoke that entitlement.
Click **Save Changes** and confirm. Changes take effect immediately for the user's next query.
### Inherited (locked) datasets
If a user has **connection-level** entitlements, all datasets in that connection are automatically included. These datasets:
* Appear with a lock icon in the permissions table
* Show as **(locked)** on their selection pill
* Cannot be individually deselected until the connection entitlement is removed
Connection entitlements are typically managed through connection-level permission settings rather than the per-user dataset table.
### Viewing a user's access
Click the **View** action on any user to open a read-only summary of their connections, datasets, groups, and role. Super Admins also see a **JSON** tab with the raw permissions payload.
## Removing entitlements
To revoke dataset or group access:
1. Open **Data Permissions** for the user.
2. Deselect the dataset or group row in the table, or click the remove button on its pill.
3. Click **Save Changes**.
Removing a dataset entitlement also removes any fine-grained permission rules attached to that dataset. See [Fine-Grained Permissions](/administration/fine-grained-permissions) for details.
Deselecting a dataset group removes the group entitlement but does not affect individually assigned datasets within that group unless you deselect those rows separately.
## Deactivating users
Deactivation removes a user from the active Liberator permissions system. The user account is disabled and the user's permissions row is deleted on the backend.
### Single user
Click the delete icon in the **Actions** column, or open **Data Permissions** and click the trash icon in the modal header.
Confirm the deactivation. The user moves to the **Deactivated Users** tab in the Add Users dialog.
### Bulk deactivation
Super Admins can deactivate multiple users at once:
1. Select users using the checkboxes in the rightmost column.
2. Click **Deactivate Selected**.
3. Type `DEACTIVATE` to confirm.
You cannot deactivate your own account. Bulk deactivation requires Super Admin role.
## Re-activating users
1. Click **Add Users** on the Users page.
2. Switch to the **Deactivated Users** tab.
3. Select the users to restore and click **Re-activate**.
4. Re-assign any dataset entitlements and fine-grained rules that were removed during deactivation.
## Verifying entitlements (end user)
Users can confirm their access from Python:
```python theme={null}
# Datasets the user is entitled to query
liberator.datasets(entitled=True)
# All datasets (including those without access)
liberator.datasets(entitled=False)
```
See [Checking Dataset Access](/python-guide/dataset-access) for schema and detail commands.
## Common tasks
| Task | Where | Required role |
| --------------------------------- | ---------------------------------------------- | -------------------- |
| Create a new account | Users → **Create User** | Super Admin |
| Combined create and enroll (2.4+) | Users → **Create User** | Super Admin |
| Invite via email (2.3+) | Users → **Create User** → **Invite via email** | Super Admin |
| Send password-reset email (2.3+) | Users → **Send reset email** | Super Admin |
| Enroll an existing account | Users → **Add Users** | Admin or Super Admin |
| Grant dataset access | Users → **Data Permissions** | Super Admin |
| Revoke dataset access | Users → **Data Permissions** → deselect | Super Admin |
| Change platform role | Users → **Edit** | Super Admin |
| Deactivate a user | Users → delete action | Super Admin |
| Re-activate a user | Users → **Add Users** → Deactivated tab | Admin or Super Admin |
# Administration Overview
Source: https://knowledge.cloudquant.com/administration/overview
Manage users, dataset entitlements, and fine-grained access controls in the CloudQuant Data Liberator admin portal
# Administration overview
The CloudQuant Data Liberator admin portal lets organization administrators control who can access the platform and which datasets each user can query. Access is managed through **entitlements** — explicit grants to datasets, dataset groups, and connections — with optional **fine-grained permissions** that restrict query scope.
## Who can manage access
The admin portal uses a role hierarchy. Each role can assign roles at or below its own level.
| Role | Description |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **User** | Query entitled datasets. No admin portal access. |
| **Admin** | Manage connections, datasets, and user groups. Can add existing user accounts to the permissions system. |
| **Super Admin** | Full platform administration — create users, edit roles, assign entitlements, configure fine-grained permissions, and deactivate users. |
Most user and entitlement management actions described in this section require **Super Admin**. Admins can view the Users page and add users to Liberator, but editing user details, assigning dataset entitlements, and deactivating users are Super Admin actions.
## Permission model
Access is layered:
1. **Platform role** (`user`, `admin`, `super_admin`) — controls what a person can do in the admin portal.
2. **Connection entitlements** — grant access to all datasets within a connection. Datasets inherited from a connection appear as locked in the permissions UI.
3. **Dataset entitlements** — grant access to individual datasets at **Editor** or **User** level.
4. **Dataset group entitlements** — grant access to a curated bundle of datasets defined in **Dataset Groups**.
5. **Fine-grained permissions** — optional per-dataset or per-group restrictions on date range, query day span, query type, row limits, keys, and columns.
```mermaid theme={null}
flowchart TD
A[User account] --> B[Platform role]
A --> C[Connection entitlements]
A --> D[Dataset entitlements]
A --> E[Dataset group entitlements]
D --> F[Fine-grained permissions]
E --> F
C --> G[Inherited dataset access]
```
### Editor vs user
For datasets and connections, entitlements are assigned at one of two levels:
| Level | Meaning |
| ---------- | ---------------------------------------------------------------------------------------------------------------------- |
| **Editor** | Full access to query the dataset (or all datasets in a connection). |
| **User** | Query access at the usage level. In the UI this is labeled **User**; internally it maps to the usage entitlement tier. |
When a user has connection-level access, all datasets in that connection are inherited automatically. Those datasets cannot be individually removed until the connection entitlement is revoked.
## Admin portal navigation
Super Admins and Admins see these relevant sections in the sidebar:
| Page | Purpose |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Users** | Add users, assign entitlements, configure fine-grained permissions |
| **User Groups** | Manage groups of users for bulk entitlement assignment |
| **Dataset Groups** | Manage bundles of datasets for bulk entitlement assignment |
| **Connections** | Configure data source connections |
| **Datasets** | View and manage individual datasets. In Liberator **2.4** and later, catalog cards show date range, symbol count, status, and last updated. Administrators can configure how often symbol discovery refreshes. |
## Related guides
Describe a source in plain language and auto-fill the Create Dataset wizard
Natural-language usage, user, and entitlement reports in the Admin UI
Super-Admin per-query usage dataset (`usage_log`) via the standard query API
Add, create, deactivate users and grant or revoke dataset access
Restrict query date ranges, row limits, keys, and columns per user
Public vs private datasets and the Publicly Available toggle
Schedule warm Parquet caches in S3 or GCS before users query
Lookup-table enrichment with translate=true
Authentication, network security, and the end-user view of entitlements
How users verify their entitled datasets from Python
# Query Usage Log
Source: https://knowledge.cloudquant.com/administration/usage-log
Query per-query usage records as a Super-Admin-only Liberator dataset
# Query usage log
In Liberator **2.4** and later, Super Admins can query **`usage_log`** — a standard Liberator dataset of **one row per query per dataset**. Use it to see who queried which dataset, which `as_of` / `back_to` window they requested, how many rows came back, and whether the query succeeded or was rejected.
This is the per-query drill-down that monthly usage totals and [AI-based reporting](/administration/ai-based-reporting) do not provide. It uses the same `liberator.query` API as any other dataset, including Python, REST, Excel, and MCP.
`usage_log` is **Super Admin only**. It is private and hidden from the standard user catalog. Regular users cannot discover or query it, and it is not granted through ordinary Data Permissions.
## Who can query it
| Role | Can query `usage_log` |
| -------------------- | --------------------- |
| **Super Admin** | Yes |
| **Admin** / **User** | No |
If a Super Admin query returns "not entitled" or the dataset is missing, usage logging may not be enabled on that instance. Contact CloudQuant.
## Query the log
The **query parameters** `as_of` and `back_to` select **when the usage occurred** (when someone ran a query). They are not the date range those users requested — those values are columns on each row.
The **symbol / key** is the **dataset that was queried**, not a ticker. `symbols='daily_bars'` returns usage rows for queries against `daily_bars`.
```python theme={null}
import liberator
# Every query on 2026-08-28 (daily granularity)
df = liberator.get_dataframe(liberator.query(
name='usage_log',
back_to='2026-08-28',
as_of='2026-08-29',
))
# Only queries against one dataset
df = liberator.get_dataframe(liberator.query(
name='usage_log',
symbols='daily_bars',
back_to='2026-08-28',
as_of='2026-08-29',
))
# One user's queries in August
df = liberator.get_dataframe(liberator.query(
name='usage_log',
back_to='2026-08-01',
as_of='2026-09-01',
where="user_id = $$jane.doe$$",
))
# Rejections and errors
df = liberator.get_dataframe(liberator.query(
name='usage_log',
back_to='2026-08-28',
as_of='2026-08-29',
where="status != $$success$$",
))
```
Recent queries are included as they complete — you do not wait for a daily rollup.
## Record grain
One row per **(query, dataset)**. A request that names several datasets produces several rows that share the same `qid`.
Rejected queries still appear, with `rows_returned` of `0` and `status` of `not_entitled`, `limit_exceeded`, or `error`.
## Columns
| Column | Meaning |
| -------------------- | ----------------------------------------------------------------------- |
| `timestamp` / `muts` | When the usage record was stored (Liberator's usual time axis) |
| `symbol` | Key field — the dataset that was queried (same value as `dataset_name`) |
| `dataset_name` | Dataset the user requested |
| `user_id` | Liberator username that ran the query |
| `system` | Client / system identifier from the request (for example `API`) |
| `as_of` | `as_of` the caller passed (the original query's end time) |
| `back_to` | `back_to` the caller passed (the original query's start time) |
| `rows_returned` | Rows delivered for that dataset; `0` if the query was rejected |
| `status` | Outcome — see below |
| `qid` | Query id; shared by every dataset row from the same request |
| `request_id` | Optional correlation id from the connection |
| `detail` | Extra reason text (for example an error class, or a client disconnect) |
| `query_timestamp` | UTC microseconds when the original query finished |
Do not confuse the **columns** `as_of` / `back_to` (what the user asked for) with the **parameters** `as_of` / `back_to` on your `usage_log` query (which slice of the log you want).
### Status values
| `status` | Meaning |
| ---------------- | ------------------------------------------------------------------------------- |
| `success` | Query completed and returned data (including an empty result that was allowed) |
| `limit_exceeded` | Rejected by a usage or entitlement cap (for example max rows or max query days) |
| `not_entitled` | Caller is not entitled to the dataset |
| `error` | Any other failure |
## Retention
Records are kept for **at least 30 days**. Retention is configured for the instance — contact CloudQuant to change it.
## Related
Aggregated usage, user, and entitlement reports in the Admin UI
Cluster health, long queries, and license utilization
Caps that show up as `limit_exceeded` on usage\_log
Liberator 2.4 release notes
# Dataset Metadata
Source: https://knowledge.cloudquant.com/api-reference/concepts/dataset-metadata
Fetch concordance-backed row counts, symbol counts, and date spans with liberator.metadata() (POST /liberator/metadata).
# Dataset metadata
Use `liberator.metadata()` to fetch **concordance-backed metrics** for one or more datasets — row counts, symbol counts, and start/end timestamps — without downloading source rows.
This is a Liberator **2.3+** API. It calls `POST /liberator/metadata` and is separate from `liberator.query()` / `liberator.get_dataframe()`. Do not pass `summary` or `symbol` to `query`; those parameters belong only on `metadata`.
The Python SDK helper ships with Liberator 2.3 client packages. Download the latest `liberator.py` from your Liberator Profile page if your client does not yet expose `liberator.metadata`.
## Parameters
| Parameter | Description | Type | Default |
| --------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------- | --------------------------- |
| `name` | Dataset name, or a list of dataset names | String or list | — (required) |
| `summary` | Collapse the per-day dimension into a whole-history summary | Boolean | `false` |
| `symbol` | Return per-symbol grain instead of per-dataset totals | Boolean | `false` |
| `symbols` | Restrict metrics to this symbol or list of symbols (also used for entitlement checks) | String or list | `null` |
| `back_to` | Beginning of the data-time window | Date string, microseconds UTC, or `null` (no lower bound) | `null` |
| `as_of` | End of the data-time window | Date string, microseconds UTC, or `null` (now) | `null` |
| `user` | Your CloudQuant-assigned user identifier | String | from `liberator.json` / env |
| `token` | Your authentication token | String | from `liberator.json` / env |
## Grain matrix (`summary` × `symbol`)
| `summary` | `symbol` | Result grain |
| --------- | -------- | ------------------------- |
| `false` | `false` | Per day, dataset totals |
| `false` | `true` | Per day, per symbol |
| `true` | `false` | Whole-dataset totals |
| `true` | `true` | Whole-history, per symbol |
## Basic example
```python theme={null}
import liberator
liberator.url = "https://api.cloudquant.ai"
metrics = liberator.metadata(
name="daily_bars",
summary=False,
symbol=False,
)
# list of metric dicts, for example:
# [{"dataset": "...", "date": "...", "symbol_count": ..., "row_count": ..., "start": ..., "end": ...}, ...]
print(metrics)
```
## Whole-dataset summary
```python theme={null}
metrics = liberator.metadata(
name="daily_bars",
summary=True,
symbol=False,
)
```
## Per-symbol metrics
```python theme={null}
# Per day + per symbol
daily_by_symbol = liberator.metadata(
name="daily_bars",
summary=False,
symbol=True,
symbols=["AAPL", "MSFT"],
)
# Whole-history per symbol
history_by_symbol = liberator.metadata(
name="daily_bars",
summary=True,
symbol=True,
symbols=["AAPL"],
)
```
## Multiple datasets
```python theme={null}
metrics = liberator.metadata(
name=["daily_bars", "minute_bars"],
summary=True,
symbol=False,
)
```
## Windowed metadata
```python theme={null}
metrics = liberator.metadata(
name="daily_bars",
summary=False,
symbol=False,
back_to="2025-01-01",
as_of="2025-01-31",
)
```
## Return shape
`liberator.metadata()` returns a **list of metric dictionaries** (the `metrics` array from the streamed response). Typical fields include:
| Field | Meaning |
| --------------- | ----------------------------------------------- |
| `dataset` | Dataset name |
| `date` | Calendar day (present when `summary=False`) |
| `symbol` | Symbol key (present when `symbol=True`) |
| `symbol_count` | Distinct symbols in the grain (when applicable) |
| `row_count` | Row count for the grain |
| `start` / `end` | Timestamp bounds for the grain |
The HTTP response is streamed NDJSON (heartbeat frames, then a final `{"metrics": [...]}` object). The Python helper consumes heartbeats and returns only the metrics list.
## Related
* [Listing datasets](/api-reference/concepts/listing-datasets) — catalog, schema, and entitlement discovery via `liberator.datasets`
* [Query parameters](/api-reference/concepts/query-parameters) — row-level queries via `liberator.query` (not metadata)
* [Checking dataset access](/python-guide/dataset-access) — Python guide for exploring entitled datasets
# Example Queries
Source: https://knowledge.cloudquant.com/api-reference/concepts/example-queries
Practical examples of CloudQuant Data Liberator queries including last known value, time series, all symbols, live datasets, and stats.
# Example queries
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `--cert-type P12 --cert liberator.pfx` flags shown in the curl examples below are only needed for releases prior to 2.0.
## Last known value
A query without date parameters retrieves the most recent available data.
```python Python theme={null}
import liberator
res = liberator.query(name='daily_bars')
df = liberator.get_dataframe(res)
df[['symbol', 'timestamp', 'open', 'high', 'low', 'close', 'volume']]
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"}
});
var df = liberator.get_dataframe(res);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
```
```javascript JavaScript theme={null}
let params = {
symbols: "AAPL",
name: "daily_bars"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Liberator liberator = new Liberator();
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
}});
for (Liberator.QueryResult res : liberator.Generate()) {
Object data = res.GetData();
if (data instanceof org.apache.arrow.vector.VectorSchemaRoot) {
printRecordBatch((org.apache.arrow.vector.VectorSchemaRoot) data);
} else if (data instanceof javax.json.JsonValue) {
System.out.println((javax.json.JsonValue) data);
}
}
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":["AAPL","IBM"],"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","debug_stream":false}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"name", "daily_bars"s}
}));
(void)arrow::PrettyPrint(*table, arrow::PrettyPrintOptions(0, 1), &std::cout);
```
Rather than displaying the entire result, you can select specific columns or pass a `fields` parameter to include only the data you need. Some key fields are returned automatically.
## Time series queries
Two date parameters control historical data retrieval:
* **`as_of`** — Retrieve data from any past date showing information as known on that date. Defaults to current time. Format: `YYYY-MM-DD HH:MM:SS` (time optional).
* **`back_to`** — Specifies where the returned dataset begins. Reads all data "back to" the specified date. Format: `YYYY-MM-DD HH:MM:SS` (time optional).
If `as_of` and `back_to` are identical, the result will not be a time series.
```python Python theme={null}
df = liberator.get_dataframe(liberator.query(
name='daily_bars',
as_of='2020-11-15',
symbols=['FB', 'AAPL', 'NFLX', 'GOOG', 'MSFT', 'IBM']))
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
}
```
```javascript JavaScript theme={null}
let params = {
back_to: "2021-01-12 00:00:00",
as_of: "2021-01-12 23:59:59",
symbols: "AAPL",
name: "daily_bars"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("back_to", "2020-01-01");
put("symbols", "AAPL");
}});
for (Liberator.QueryResult res : liberator.Generate()) {
Object data = res.GetData();
if (data instanceof org.apache.arrow.vector.VectorSchemaRoot) {
printRecordBatch((org.apache.arrow.vector.VectorSchemaRoot) data);
} else if (data instanceof javax.json.JsonValue) {
System.out.println((javax.json.JsonValue) data);
}
}
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
symbols="AAPL",
as_of="2019-09-15",
back_to="2019-09-10"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","back_to":"2022-12-01","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","debug_stream":false}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
{"as_of", "2021-04-10"s},
{"back_to", "2021-03-10"s}
}));
```
## Query all symbols
Omit the `symbols` parameter to retrieve all symbols in a dataset.
```python Python theme={null}
df = liberator.get_dataframe(liberator.query(
name='daily_bars',
as_of='2020-11-15'))
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
}});
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"name":"daily_bars","as_of":"2020-11-15","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query
```
## Query a live dataset
Set the `as_of` parameter to `"live"` for real-time data streaming.
```python Python theme={null}
res = liberator.query(name='live_nasdaq_composite', as_of='live')
for batch in res:
df = batch.to_pandas()
print(df)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "live_nasdaq_composite"},
{"as_of", "live"},
{"symbols", "AAPL"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
}
```
```cpp C++ theme={null}
Liberator::Result ptr = liberator.query({
{"symbols", std::vector{"AAPL"}},
{"name", "live_nasdaq_composite"s},
{"as_of", "live"s}
});
auto generator = *std::get_if(&ptr);
for (auto res = generator(); res.index(); res = generator())
{
auto batch = *std::get_if<2>(&res);
(void)arrow::PrettyPrint(*batch, arrow::PrettyPrintOptions(0, 1), &std::cout);
}
```
## Query stats as JSON
Setting the `stats` parameter to `"total"` returns a JSON result with the count per symbol instead of data rows.
```python Python theme={null}
res = liberator.query(
name='daily_bars',
as_of='2021-04-10',
back_to='2021-03-10',
symbols=['ES'],
stats='total'
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"},
{"stats", "total"}
});
Console.WriteLine(JsonDocumentToIndentedString(res));
```
```r R theme={null}
print(liberator::query(
symbols="AAPL",
name="daily_bars",
as_of="2019-09-15",
back_to="2019-09-10",
stats="total"
))
```
## Get DataFrame / table
Convert query results into a tabular format native to your language.
```python Python theme={null}
df = liberator.get_dataframe(
liberator.query(name='daily_bars', as_of='2020-11-15', symbols=['AAPL'])
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
var df = liberator.get_dataframe(res);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
```
```r R theme={null}
res = liberator::get_dataframe(liberator::query(
symbols="AAPL",
name="daily_bars",
as_of="2025-01-01",
back_to="2024-12-01"
))
print(res)
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
}));
(void)arrow::PrettyPrint(*table, arrow::PrettyPrintOptions(0, 1), &std::cout);
```
# Listing Datasets
Source: https://knowledge.cloudquant.com/api-reference/concepts/listing-datasets
How to list, filter, and inspect available datasets using the CloudQuant Data Liberator API across all supported languages.
# Listing datasets
The `datasets` function retrieves a JSON-formatted list of all datasets available in CloudQuant Data Liberator, with optional filtering and metadata.
## Parameters
| Parameter | Description | Type | Default |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------- |
| `entitled` | Only retrieve datasets you have access to | Boolean | `false` |
| `schema` | Include dataset descriptions and metadata | Boolean | `false` |
| `details` | Include column-level information (names, descriptions). In Liberator **2.4+**, also includes catalog-card fields: date range, symbol count, status, and last updated. Requires `schema` to also be set. | Boolean | `false` |
| `user` | Your CloudQuant-assigned user identifier | String | — |
| `token` | Your assigned authentication token | String | — |
## List all datasets
```python Python theme={null}
import liberator
res = liberator.datasets()
print(res)
```
```csharp C# theme={null}
Liberator liberator;
var res = liberator.datasets();
Console.WriteLine(JsonDocumentToIndentedString(res));
```
```javascript JavaScript theme={null}
let dataset_params = {
schema: false,
details: true
};
liberator.datasets(dataset_params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Liberator liberator = new Liberator();
Object datasets_id = liberator.datasets(new HashMap<>());
for (Liberator.QueryResult res : liberator.Generate()) {
Object data = res.GetData();
if (data instanceof javax.json.JsonValue) {
System.out.println((javax.json.JsonValue) data);
}
}
```
```r R theme={null}
print(liberator::datasets())
```
```cpp C++ theme={null}
Liberator liberator;
auto ptr = liberator.datasets({});
auto json = *std::get_if<0>(&ptr);
if (json)
{
rapidjson::StringBuffer buf;
rapidjson::Writer writer(buf);
json->Accept(writer);
std::cout << buf.GetString() << std::endl;
}
```
### Example output
```json theme={null}
{
"Intraday": ["SSR_Forward_DSI", "Halt_Production_DSI", "SSR_Production_DSI", "News", "Twitter", "Stocktwits"],
"Monthly": ["Joblink Ticker Mapping", "VerticalKnowledge Indeed"],
"Market Data": ["daily_bars", "minute_bars", "nbbo", "trades", "daily_bars_adjusted"]
}
```
## List entitled datasets only
Filter results to show only datasets matching your access permissions.
```python Python theme={null}
liberator.datasets(entitled=True)
```
```csharp C# theme={null}
var res = liberator.datasets(new Dictionary() {
{"entitled", true}
});
```
```java Java theme={null}
Object datasets_id = liberator.datasets(new HashMap<>() {{
put("entitled", true);
}});
```
```r R theme={null}
print(liberator::datasets(entitled=TRUE))
```
```cpp C++ theme={null}
liberator.datasets({{"entitled", true}});
```
## Include schema and column details
```python Python theme={null}
# Schema only
liberator.datasets(schema=True)
# Schema with column details (2.4+: also date range, symbol count, status, last updated)
liberator.datasets(schema=True, details=True)
```
```csharp C# theme={null}
var res = liberator.datasets(new Dictionary() {
{"schema", true},
{"details", true}
});
```
```r R theme={null}
print(liberator::datasets(schema=TRUE, details=TRUE))
```
```cpp C++ theme={null}
liberator.datasets({{"schema", true}, {"details", true}});
```
## Related
For concordance-backed row counts, symbol counts, and date spans without downloading source rows, see [Dataset metadata](/api-reference/concepts/dataset-metadata) (`liberator.metadata`, Liberator 2.3+).
In Liberator **2.4** and later, `liberator.datasets(schema=True, details=True)` also returns catalog-card health (date range covered, symbol count, status, last updated) so you can check freshness before you query. The same fields appear on dataset cards in the Liberator catalog UI.
# Queries & Working with Large Datasets
Source: https://knowledge.cloudquant.com/api-reference/concepts/queries-large-datasets
Best practices for querying large datasets with the CloudQuant Data Liberator API, including point-in-time and time series query patterns.
# Queries & working with large datasets
The CloudQuant Data Liberator service frequently returns extensive result sets, sometimes reaching millions of rows. These queries can be time-consuming to execute. Since data frequencies vary across datasets, start with narrow timeframes for specific symbols before expanding scope.
Consider running a point-in-time query for one symbol to get an idea of how large your dataset is prior to running other queries.
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `--cert-type P12 --cert liberator.pfx` flags shown in the curl examples below are only needed for releases prior to 2.0.
## Query types
Most dataset queries fall into two categories:
* **Point-in-time queries**
* **Time series queries**
Familiarity with both query types early on will improve your API usage efficiency.
## Point-in-time queries
Omit the `back_to` parameter to receive single point-in-time data for each specified symbol based on the `as_of` date/time. If you also exclude `as_of`, CloudQuant Data Liberator defaults to the current date/time.
```python Python theme={null}
df = liberator.get_dataframe(
liberator.query(
name='daily_bars',
as_of='2020-11-15',
symbols=['AAPL']
)
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2020-11-15"},
{"symbols", "AAPL"}
});
```
```javascript JavaScript theme={null}
let params = {
symbols: "AAPL",
name: "daily_bars",
as_of: "2020-11-15"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("symbols", "AAPL");
}});
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
as_of="2020-11-15",
symbols="AAPL"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","as_of":"2020-11-15","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL"}},
{"name", "daily_bars"s},
{"as_of", "2020-11-15"s}
}));
```
## Time series queries
Include a `back_to` parameter that precedes your `as_of` value. When `as_of` is excluded, it defaults to the present moment.
```python Python theme={null}
df = liberator.get_dataframe(
liberator.query(
name='daily_bars',
as_of='2020-11-15',
back_to='2018-11-15',
symbols=['FB', 'AAPL', 'NFLX', 'GOOG', 'MSFT', 'IBM']
)
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
}
```
```javascript JavaScript theme={null}
let params = {
back_to: "2021-01-12 00:00:00",
as_of: "2021-01-12 23:59:59",
symbols: "AAPL",
name: "daily_bars"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("back_to", "2020-01-01");
put("symbols", "AAPL");
}});
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
symbols="AAPL",
as_of="2019-09-15",
back_to="2019-09-10"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","back_to":"2022-12-01","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
{"as_of", "2021-04-10"s},
{"back_to", "2021-03-10"s}
}));
```
When using `back_to` or `as_of` parameters, the time component is always used even if you do not specify it. Therefore, if you say `as_of: "2023-01-15"`, you are actually saying `as_of: "2023-01-15 00:00:00"`. This may affect result precision depending on your data requirements.
# Query Parameters Reference
Source: https://knowledge.cloudquant.com/api-reference/concepts/query-parameters
Complete reference for all parameters accepted by the CloudQuant Data Liberator query function across all SDKs.
# Query parameters reference
This page documents the parameters accepted by the `query` function across all CloudQuant Data Liberator SDKs. Parameter names are consistent across languages; only the types and calling conventions differ.
At least one of the following is required for every query:
* A dataset via `name` **or** a raw `sql` statement.
* A time anchor: `back_to`, `as_of`, `max_lookback` (LKV mode), or `record_limit` (N-Query mode).
Unknown parameters are rejected by the server when strict validation is enabled (the default).
## Time range parameters
| Parameter | Default | Description |
| -------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `as_of` | `None` | End time for the query. `None` means "now". Integer values are interpreted as microseconds since epoch; strings are interpreted as datetimes. In N-Query backward mode the boundary is **exclusive** (`muts < as_of`). |
| `back_to` | `None` | Start time for the query. `None` means as-of only (single point in time, or LKV). In N-Query forward mode the boundary is **inclusive** (`muts >= back_to`). |
| `max_lookback` | (from dataset config) | Used for LKV (last-known-value) queries. Integer values **less than 1,000,000** are treated as a number of partitions to look back; values **greater than or equal to 1,000,000** are treated as a duration in microseconds. May be supplied per-query or inherited from dataset config. |
For LKV mode, omit `back_to` and use `as_of` (or the default "now") together with `max_lookback`. For N-Query mode, provide either `as_of` (backward) or `back_to` (forward) along with `record_limit`.
### Date format
Datetime strings use the format `YYYY-MM-DD HH:MM:SS` (the time portion is optional). When the time portion is omitted, `00:00:00` is assumed — for example, `as_of: "2023-01-15"` is interpreted as `as_of: "2023-01-15 00:00:00"`. If `as_of` and `back_to` are identical, the result is a single point rather than a time series.
## Dataset & symbol parameters
| Parameter | Default | Description |
| ----------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `None` | Dataset name(s). A **string** selects a single dataset; a **list of strings** triggers a superquery (multi-dataset merge). Required unless `sql` is provided. |
| `symbols` | `None` | Symbol(s) to filter on. Accepts a string, a list of strings, or `None` for all symbols in the dataset. In Liberator **2.4+**, an unescaped `%` in a value is a `LIKE` wildcard (for example `'PWR%'`). Mix exact keys and patterns in one list. Escape a literal percent as `\%`. |
| `keys` | `None` | Synonym for `symbols`. |
| `fields` | `None` | Field names to return. List of strings or `None` for all fields. Mandatory fields cannot be removed. |
| `distinct` | `["symbol"]` | Partition key for LKV queries: the columns used in `PARTITION BY` (for example, `["symbol"]` or `["symbol", "dataset"]`). |
| `translate` | `None` | When set to `true` or `false`, enables or disables [dataset field mapping](/administration/dataset-field-mapping) lookups configured on the dataset. |
| `where` | `None` | Optional SQL predicate appended to generated filters with `AND`. Omit the `WHERE` keyword — supply only the condition (e.g. `"volume > 1000000"`). Not compatible with `sql`. |
| `sql` | `None` | Raw SQL `SELECT` statement. When present, replaces parameter-based query construction (`name`, time range, `where`, etc.). See [Raw SQL](#raw-sql) below. |
## Custom WHERE filters
The `where` parameter adds extra predicates on top of the time-range and symbol filters CloudQuant Data Liberator generates automatically. The value is appended as `AND ()` to the internal query.
| Aspect | Detail |
| --------------------- | ---------------------------------------------------------------------------------------------- |
| **Syntax** | SQL expression only — no leading `WHERE` |
| **Compatible with** | `name`, `back_to`, `as_of`, `symbols`, `fields`, `record_limit`, LKV (`max_lookback`), N-Query |
| **Incompatible with** | `sql` |
| **Column quoting** | Column names matching the dataset schema are auto-quoted when required |
```python Python theme={null}
result = liberator.query(
name="daily_bars",
symbols=["AAPL"],
back_to="2024-01-01",
as_of="2024-07-01",
where="volume > 1000000"
)
```
```csharp C# theme={null}
var result = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"symbols", new[] {"AAPL"}},
{"back_to", "2024-01-01"},
{"as_of", "2024-07-01"},
{"where", "volume > 1000000"}
});
```
```r R theme={null}
result <- liberator::query(
name = "daily_bars",
symbols = c("AAPL"),
back_to = "2024-01-01",
as_of = "2024-07-01",
where = "volume > 1000000"
)
```
For multi-key datasets (e.g. concordance tables with a `dataset` column), `where` can filter on columns beyond `symbol`:
```python theme={null}
result = liberator.query(
name="concordance",
back_to="2024-01-01",
as_of="2025-01-01",
where="dataset = 'my_derived_dataset'"
)
```
## Raw SQL
The `sql` parameter accepts a complete SQL `SELECT` and bypasses automatic query construction. Use it when you need direct control over joins, subqueries, or complex predicates.
### Requirements
| Rule | Detail |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Statement type | Must be a `SELECT` (each arm of `UNION` / `INTERSECT` / `EXCEPT` is validated separately) |
| `WHERE` clause | Required on every `SELECT` arm |
| Time-range predicate | By default, `WHERE` must include `=`, `>`, `>=`, `<`, `<=`, or `BETWEEN` on `muts` or columns listed in `LIBERATOR_SQL_TIME_RANGE_COLUMNS` |
| Dataset references | Quote table names with double quotes: `"my_dataset"` |
| String literals | Use `$$value$$` dollar-quoting for symbol and string filters |
| Timestamps | Express `muts` bounds as microsecond integers |
### Parameters allowed with `sql`
When `sql` is supplied, only auth, transfer, and cache options may accompany it. Non-empty construction parameters (`name`, `back_to`, `as_of`, `where`, `fields`, etc.) raise an error.
Allowed alongside `sql`: `user`, `system`, `token`, `compress`, `json_xfer`, `batch_size`, `debug_stream`, `force_regen`, `dependent_partition_name`, `skip_validation`, `skip_discovery`, `symbols`, `symbol_key_sequences`, `data_key_column`.
Do not combine `sql` with `name`, `where`, `back_to`, `as_of`, or other query-construction parameters. Choose parameter-based queries or raw SQL, not both.
## Query limits & ordering
| Parameter | Default | Description |
| -------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `record_limit` | `None` | When set to a non-zero integer, triggers **N-Query mode**. **Negative** values (e.g. `-200`) apply a per-symbol limit. **Positive** values (e.g. `200`) apply a global clamp (most recent N records across all symbols). Ignored when both `as_of` and `back_to` are provided. |
| `order` | `None` | Result ordering. One of `"asc_strict"`, `"asc_loose"`, `"desc_strict"`, `"desc_loose"`. Default is ascending. |
### N-query mode
N-Query mode is activated by supplying a non-zero `record_limit`.
* **Anchor:** Backward N-Query uses `as_of` (exclusive); forward N-Query uses `back_to` (inclusive). Exactly one anchor is required.
* **Per-symbol vs global:** A negative `record_limit` returns up to N records per symbol; a positive value returns N records total, distributed by recency.
* **Compatibility:** N-Query works with both single-dataset and superquery queries. When both `as_of` and `back_to` are set, `record_limit` is ignored and a standard time-range query is executed.
## Superquery parameters
A **superquery** is triggered when `name` is a **list** of dataset names. It merges and resamples data from multiple datasets into a single result.
| Parameter | Default | Description |
| ---------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `superq_resample_rule` | `"1D"` | Resampling interval. Accepts pandas-style strings (e.g. `"5T"`, `"1D"`) or Liberator-native values: fixed (`1s`–`60s`, `1m`–`240m`, `1d`, `7d`) or calendar (`day`, `wk`, `mo`, `qtr`, `yr`). |
| `superq_fill_forward` | `True` | Fill forward missing values. Only `false` is currently supported when using aggregates. |
| `superq_aggregates` | `False` | Aggregation spec. `False` / `None` disables aggregation; a dict provides per-dataset record-level (`last_timestamp`, `first_timestamp`) and column-level aggregates (see below). |
### Column-level aggregate aliases
| Alias | Meaning | Description |
| ----- | ------- | --------------------------- |
| `o` | first | First value in the bucket |
| `h` | high | Maximum value in the bucket |
| `l` | low | Minimum value in the bucket |
| `c` | last | Last value in the bucket |
| `s` | sum | Sum of values in the bucket |
Record-level aggregates (`last_timestamp`, `first_timestamp`) apply to the entire dataset per bucket. Output columns are named `{dataset}_{column}_{alias}` (e.g. `trades_price_o`, `trades_volume_s`) for column-level aggregates, and `{dataset}__first_timestamp` / `{dataset}__last_timestamp` for record-level aggregates.
**Dict format example:**
```json theme={null}
{
"my_dataset": [
"last_timestamp",
"first_timestamp",
{
"price": ["o", "h", "l", "c"],
"volume": ["s"],
"datetime_utc": []
}
]
}
```
## Authentication & authorization
| Parameter | Default | Description |
| --------- | ------- | --------------------------------------------- |
| `user` | `None` | Authorized user name (used for entitlements). |
| `system` | `None` | System identifier. Typically `"API"`. |
| `token` | `None` | Authentication token. |
The `user` and `token` parameters can be provided per-query or configured once at the SDK level:
* **Python:** Set via the `liberator.json` file in your working directory.
* **JavaScript:** Use `liberator.set_default("query", credentials)` to avoid passing them every time.
* **RESTful:** Include in each request's JSON body.
* **Other SDKs:** Loaded from `liberator.json` in the working directory.
## Connection & transfer
| Parameter | Default | Description |
| ------------ | ------- | ---------------------------------------------- |
| `compress` | `False` | Compress the response on the wire. |
| `json_xfer` | `False` | Use JSON transfer format instead of Arrow IPC. |
| `batch_size` | `25000` | Rows per batch for chunked / streaming output. |
## Cache & derived data
| Parameter | Default | Description |
| -------------------------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `force_regen` | `False` | Bypass caches (e.g. snapfresh, superquery cache) and regenerate results. |
| `dependent_partition_name` | `None` | Partition UUID for dependency tracking (used for cache invalidation). |
| `skip_validation` | `None` | Tri-state override for cache\_manager validation. When unset, the server default from environment is used. |
| `skip_discovery` | `None` | Tri-state override for partition discovery. When unset, the server default from environment is used. |
## Streaming & debug
| Parameter | Default | Description |
| ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `force_streaming` | `False` | Enable incremental streaming for lower time-to-first-byte. Supported only for single-table, non–N-Query reads. |
| `debug_stream` | `False` | Enable debug / progress output. Language-specific type (e.g. `stderr()` in R, `std::ostream*` in C++, `System.IO.TextWriter` in C#). |
| `warning_stream` | `None` | Output stream for warning information. Same type conventions as `debug_stream`. |
## SDK connection options
These options configure the SDK client rather than the query itself.
| Parameter | Default | Description |
| --------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `url` | From Profile download | CloudQuant Data Liberator server URL. Preconfigured in clients downloaded from your Liberator Profile page. Override for a specific IP or port, e.g. `http://127.0.0.1:47753`. |
## Example queries
### Basic time-range query
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL", "MSFT"],
back_to="2025-01-01 00:00:00",
as_of="2025-01-02 00:00:00",
fields=["Price", "Volume"],
order="asc_strict"
)
```
### Symbol pattern filter (2.4+)
```python theme={null}
result = liberator.query(
name="energy_contracts",
symbols="PWR%", # LIKE wildcard; mix with exact keys in a list if needed
back_to="2026-08-01",
as_of="2026-08-02",
)
```
### LKV (last-known-value) query
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
max_lookback=3 # last 3 partitions
)
```
### N-query: most recent 200 records (backward from `as_of`)
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
record_limit=200,
as_of="2025-01-15 00:00:00"
)
```
### N-query: first 200 records (forward from `back_to`)
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
record_limit=200,
back_to="2025-01-01 00:00:00"
)
```
### N-query per-symbol (negative limit)
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL", "MSFT"],
record_limit=-10, # up to 10 records per symbol
as_of="2025-01-15 00:00:00"
)
```
### Superquery with aggregates
```python theme={null}
result = liberator.query(
name=["dataset1", "dataset2"],
symbols=["AAPL"],
back_to="2021-01-01",
as_of="2021-03-01",
superq_resample_rule="5T",
superq_fill_forward=False,
superq_aggregates={
"dataset1": [
"last_timestamp",
"first_timestamp",
{
"Trade Price": ["o", "h", "l", "c"],
"volume": ["s"]
}
]
}
)
```
### Raw SQL
```python theme={null}
result = liberator.query(
sql='SELECT * FROM "my_dataset" WHERE muts >= 1704067200000000 AND muts < 1704153600000000 AND symbol = $$AAPL$$',
user="my_user",
system="API"
)
```
### Parameter-based query with `where`
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2025-01-01 00:00:00",
as_of="2025-01-02 00:00:00",
where="volume > 0",
fields=["Price", "Volume"],
order="asc_strict"
)
```
# C++ SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/cpp/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator C++ API.
# C++ SDK getting started
The CloudQuant Data Liberator C++ SDK uses Apache Arrow for high-performance data streaming with a generator-based iteration pattern.
## Required security files
Place the following files from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Type reference
The SDK uses `std::variant` types for flexible argument and return handling:
```cpp theme={null}
using Arg = std::variant,
std::ostream*>;
using Func = std::function,
std::shared_ptr>()>;
using Result = std::variant, Func>;
```
## Quick start
### Query with get\_table
The simplest approach converts results directly into an Arrow Table:
```cpp theme={null}
Liberator liberator;
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
}));
(void)arrow::PrettyPrint(*table, arrow::PrettyPrintOptions(0, 1), &std::cout);
```
### Query with generator pattern
For streaming or large results, use the generator pattern to iterate through record batches:
```cpp theme={null}
Liberator liberator;
Liberator::Result ptr;
ptr = liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s}
});
auto generator = *std::get_if(&ptr);
for (auto res = generator(); res.index(); res = generator())
{
auto batch = *std::get_if<2>(&res);
(void)arrow::PrettyPrint(*batch, arrow::PrettyPrintOptions(0, 1),
&std::cout);
}
```
# C# SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/csharp/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator C# API.
# C# SDK getting started
The CloudQuant Data Liberator C# SDK streams data as Arrow Record Batches, which can be converted to Microsoft DataFrames.
## Required security files
You need the following files from your Downloads ZIP, placed in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
`liberator.json` must be in your working directory.
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Quick start
### Query and iterate record batches
```csharp theme={null}
Liberator liberator;
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DateTime.Now.ToString("hh:mm:ss.fff") + " " + DataFrameUtils.PrettyPrint(df));
}
```
### Get DataFrame directly
```csharp theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
var df = liberator.get_dataframe(res);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
```
# Configuring Your Account
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/configuring-account
How to set up your CloudQuant account credentials in the Excel Plug-In.
# Excel plug-in: configuring your account
Before accessing any data, you need account information from CloudQuant: **User ID**, **Token**, and **Base URL**.
## Getting started
Download the Excel Plug-In from your Liberator Profile page.
In Liberator **2.4** and later, the CloudQuant Excel plugin is also available from the Liberator client UI — you do not need a separately shared installer.
After installation, Excel displays a new **CloudQuant** Ribbon menu item.
## Configuration steps
1. Select the **CloudQuant** tab
2. Select the **Account** icon to open settings
3. Enter your account credentials in the dialog
## Required information
* **User ID**, **Token**, and **Base URL** — provided on your Liberator Profile page (no trailing slash on Base URL)
When you select OK, the application verifies your login credentials. Once verified, the remaining ribbon buttons are enabled so you can create, run, or interact with queries.
# Creating a Query Using the Ribbon Bar Wizard
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/ribbon-bar-wizard
Step-by-step guide to creating queries in the Excel Plug-In using the Ribbon Bar Wizard.
# Excel plug-in: creating a query using the Ribbon Bar wizard
The **Create** icon opens a Wizard that guides you through the new query process.
If the Wizard ribbon is grayed out, open the right side task pane and select "Query". The Wizard will now be active.
## Step-by-step process
### 1. Provide a query name
You can assign a name to your query at the beginning of the dialog. The system provides a default query name if you skip this step, though naming your query is considered best practice.
### 2. Select dataset from the tree
Available datasets are organized hierarchically by group. Double-click the first arrow or select the second arrow to explore group contents.
#### Searching for datasets by name
A search text box lets you locate specific datasets by name rather than browsing the tree structure.
### 3. Select symbols
Symbols are central to most queries. You have three options:
* Type in a symbol manually
* Link symbols to a spreadsheet location
* Retrieve all available symbols
If you do not specify a set of (one or more) symbols, you will receive all the symbols in the dataset.
### 4. Time range or point in time selection
This dialog allows specification of temporal parameters for the query.
Some datasets contain large amounts of data. Excel has limitations on how many rows of data can appear in a spreadsheet tab. It is best practice to query for a specific symbol first.
### 5. Select data location
Select where results should appear. The **Active Cell** option places data at your current spreadsheet location.
Adjust the active cell location by selecting a cell in the spreadsheet before selecting Finish.
### 6. View results
Query results appear as a formatted table within the spreadsheet.
# Creating a Query Using the Pop Out Task Pane
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/task-pane
How to use the Excel Plug-In task pane to create, manage, and run queries.
# Excel plug-in: creating a query using the pop out Task Pane
## The pop out Task Pane
A pop-out task pane appears on the right side of your spreadsheet and can be minimized, opened, or closed.
If the pane is not visible, use the "Restore Task Pane" button in the CloudQuant menu.
## Pane features
### Query tab
The Query tab displays queries you have created, allowing you to edit, re-run, or delete them. Queries are stored in the spreadsheet where they are created, enabling sharing with other licensed users.
The bottom section shows query results. If a query fails, it displays the reason.
### Datasets tab
The Datasets tab lets you create queries directly from the task pane as a shortcut to using the Wizard:
1. **Step 1**: Select your dataset
2. **Step 2**: Name your query, enter symbols, and select your time range
3. **Step 3**: Select data placement — "New Sheet", "Active Cell", or "Replace Table" (for editing existing queries)
When the DATASET pane is selected, ribbon bar options are grayed out. Select the QUERY tab to restore them.
### Resizing the Task Pane
Drag the left frame edge until your cursor becomes a double arrow, then click and drag to resize. Columns can also be resized using double-arrows on column dividers.
### Restoring the pane
Use the restore icon in the CloudQuant ribbon menu if you close the pane entirely.
## Result sets
Results are placed into tables named after your query. If data goes to a new sheet, that sheet is also named after the query.
Give queries specific, simple names. Access named ranges through Excel's Name Manager (accessible via the lightbulb icon).
# Understanding Queries
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/understanding-queries
Overview of point-in-time and time series query types in the Excel Plug-In.
# Excel plug-in: understanding queries
Queries with CloudQuant Data Liberator allow you to retrieve two distinct types of data: **point-in-time data** or **time series data**.
## Point-in-time data
When requesting point-in-time data, you typically query information for the current moment. However, you can specify any particular point in time to examine how data appeared at that previous moment — known as the "as of" date in CloudQuant Data Liberator terminology.
When reading a dataset with point-in-time queries, the system returns data for your specified time. This lets you retrieve historical snapshots showing what information was known at earlier dates.
Think of point-in-time data structurally: one row for each given trading symbol.
## Time series data
Time series data returns multiple rows for each trading symbol from your specified dataset. For instance, requesting Minute Bar stock prices over a ten-minute period generates 10 data rows per symbol.
# Java SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/java/getting-started
Prerequisites and setup guide for using the Java CloudQuant Data Liberator External API.
# Java SDK getting started
The CloudQuant Data Liberator Java SDK uses Apache Arrow for efficient data streaming and provides a generator-based iteration pattern.
## Prerequisites
```bash Debian/Ubuntu theme={null}
sudo apt-get install ca-certificates-java openjdk-17-jdk openjdk-17-jre maven
```
```bash RHEL/CentOS theme={null}
sudo dnf install ca-certificates java-17-openjdk java-17-openjdk-devel maven
```
Run the `./build_maven.sh` script to build the Maven project and install the project dependencies.
## Required security files
Place the following files from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
`liberator.json` must be in your working directory.
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Quick start
```java theme={null}
Liberator liberator = new Liberator();
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("symbols", "AAPL");
}});
for (Liberator.QueryResult res : liberator.Generate()) {
Object res_id = res.GetId();
Object data = res.GetData();
if (data instanceof org.apache.arrow.vector.VectorSchemaRoot) {
org.apache.arrow.vector.VectorSchemaRoot recordBatch =
(org.apache.arrow.vector.VectorSchemaRoot) data;
printRecordBatch(recordBatch);
} else if (data instanceof javax.json.JsonValue) {
javax.json.JsonValue jsonValue = (javax.json.JsonValue) data;
System.out.println(jsonValue);
}
}
```
# JavaScript SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/javascript/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator NodeJS module.
# JavaScript SDK getting started
The CloudQuant Data Liberator NodeJS module provides promise-based access to CloudQuant datasets.
## Installation
Install the `liberator_module` package in your NodeJS project.
## Required security files
Place the following file from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
## Quick start
```javascript theme={null}
const liberator = require('liberator_module').liberator;
const fs = require('fs');
let credentials = {
user: JSON.parse(fs.readFileSync('liberator.json')).user,
token: JSON.parse(fs.readFileSync('liberator.json')).token,
};
liberator.set_default("query", credentials);
let params = {
as_of: "2021-01-12 23:59:59",
back_to: "2021-01-12 00:00:00",
symbols: "AAPL",
name: "daily_bars",
};
liberator.query(params).then(function(results) {
console.log(results);
});
```
Different datasets have varying update frequencies. Some datasets (like earnings reports) update quarterly, while others update intraday. Time series queries may return limited results depending on data availability.
# Advanced - Query Data Using requests.post RESTful API
Source: https://knowledge.cloudquant.com/api-reference/python/advanced-rest-api
How to query CloudQuant Data Liberator data directly using Python's requests.post method with the RESTful API.
# Advanced - query data using requests.post RESTful API
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `cert=cert` argument and `pfx_to_pem` helper shown below are only needed for releases prior to 2.0.
## requests.post from Python to access CloudQuant Data Liberator data
`requests.post` is an HTTP method you can use to send query arguments to the CloudQuant Data Liberator server for processing.
## Post arguments
* **`{base_url}/liberator/query`** — The query endpoint. Use the base URL from your Liberator Profile page download.
* **`data=json.dumps(...)`** - This `json.dumps` section provides the body of the query with these parameters:
* `compress`
* `json_xfer`
* `user`
* `token`
* `name`
* `as_of`
* `back_to`
* `symbols`
* `system`
* **`headers={'Content-Type':'application/json'}`**
* **`cert=cert`** - cert comes from `with pfx_to_pem('liberator.pfx','') as cert:`
## Query the last known value(s)
To get the last known value in any dataset, do not provide the `as_of` or `back_to` arguments. This will give you the last known value for the given dataset and symbols.
## Query for a time series result
Adding the `back_to` argument for any query will give you the time series data all the way back to the specified date.
Every dataset has different data frequencies. Use short time frames until you become familiar with the datasets. Some datasets are quite large, and an over-extended request can return an extremely large amount of data.
## json.dumps components
| Argument | Description | Type | Example |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ------------------------------------ |
| `symbols` | The security trading symbol(s) you wish to query | String, or List | `"symbols":["AAPL","TLT","GOOG"]` |
| `name` | The name of the dataset **(Required)** | String | `"name": "daily_bars"` |
| `as_of` | This value can be any past date so that you can see the data as it was known on the "as of" date. Defaults to now. | String `YYYY-MM-DD HH:MM:SS` (HH:MM:SS optional) | `"as_of":"2020-11-22 19:51:31"` |
| `back_to` | The date where the return dataset should begin. | String `YYYY-MM-DD HH:MM:SS` (HH:MM:SS optional) | `"back_to":"2020-01-01"` |
| `url` | Optional. The URL of the CloudQuant Data Liberator server. Preconfigured in clients downloaded from your Liberator Profile page. | String | `'http://127.0.0.1:47753'` |
| `system` | The name of the authorized system from which you are querying. | String | `"system":"API"` |
| `compress` | The data compression method on the wire. | Boolean | `True` or `False` |
| `json_xfer` | JSON transfer. Usually False. | Boolean (Always False) | `"json_xfer":False` |
| `user` | The user identifier (as assigned by CloudQuant) | String | `"user":"myUserID"` |
| `token` | The user's assigned token | String | `"token":"mypersonal-private-token"` |
## Example: calling the CloudQuant Data Liberator REST API with Python
```python theme={null}
import sys
import zlib
import json
import base64
import requests
import tempfile
import contextlib
import pyarrow as pa
import OpenSSL.crypto
@contextlib.contextmanager
def pfx_to_pem(pfx_path, pfx_password):
'''Decrypts the .pfx file to be used with requests.'''
with tempfile.NamedTemporaryFile(suffix='.pem') as t_pem:
t_pem.close()
f_pem = open(t_pem.name, 'wb')
pfx = open(pfx_path, 'rb').read()
p12 = OpenSSL.crypto.load_pkcs12(pfx, pfx_password)
f_pem.write(OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM, p12.get_privatekey()))
f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, p12.get_certificate()))
ca = p12.get_ca_certificates()
if ca is not None:
for cert in ca:
f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert))
f_pem.close()
yield t_pem.name
base_url = '' # from Liberator Profile page download
compressed_transfer = True
# POST the query and prepare for a stream of single-line JSON replies
with pfx_to_pem('liberator.pfx', '') as cert:
r = requests.post(f'{base_url}/liberator/query',
data=json.dumps({"compress": compressed_transfer, "json_xfer": False,
"user": user, "token": token,
"name": name, "as_of": as_of, "back_to": back_to, "symbols": symbols,
"system": "API"}),
headers={'Content-Type': 'application/json'},
cert=cert,
stream=True)
# Ensure that the request was successful
if r.status_code != 200:
print(r.json())
else:
batches = []
for line in r.iter_lines(4096):
# Show progress
print('.', end='')
sys.stdout.flush()
# The response stream contains informational messages other than "batch"
if line[0:14] == b'{"exception":"':
print('Error: ' + json.loads(''.join([chr(c) for c in line]), encoding='latin-1')['value'])
break
if line[0:10] != b'{"batch":"':
continue
# Cut out the payload from the JSON line without json module overhead
decoded = base64.b64decode(line[10:-3])
# Interpret the payload as an Apache Arrow IPC stream
reader = pa.ipc.open_stream(zlib.decompress(decoded) if compressed_transfer else decoded)
# Accumulate RecordBatch objects in a list
batches.extend([batch for batch in reader])
print('')
if batches:
# Create an Arrow Table view on the batches and convert to a pandas DataFrame
df = pa.Table.from_batches(batches).to_pandas()
if 'muts' in df and '_seq' in df:
# By default, the data is sorted by symbol... re-sort by time
df = df.sort_values(['muts', '_seq'])
df.reset_index(drop=True, inplace=True)
print(df)
```
# CloudQuant Charting
Source: https://knowledge.cloudquant.com/api-reference/python/charting
Python library for creating branded financial visualizations including candlestick charts, histograms, line charts, bar charts, and more.
# CloudQuant charting
CloudQuant Charting is a Python library for creating branded financial visualizations using the CloudQuant Data Liberator API.
**Import statement:**
```python theme={null}
from liberator import charting
```
## Chart types
### Histogram
Generates distribution/histogram charts with CloudQuant branding.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame containing data
* `col` - string name of column to visualize
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - support HTML tags like ``, ``, ` `
* `histnorm` - normalization type: `'percent'`, `'probability'`, `'density'`, or `'probability density'`
* `width`, `height` - dimensions in pixels (min 400px)
* `overlay_df`, `overlay_col` - add comparison series
**Example:**
```python theme={null}
import liberator
from liberator import charting
res = liberator.query(name="daily_bars")
df_bars = liberator.get_dataframe(res)
df_bars = df_bars.sort_values(by=['volume'], ascending=[False])
change = df_bars.close - df_bars.open
df_bars["change"] = change
fig = charting.Histogram(
df=df_bars[0:100],
col="change",
title="Change Px Distribution of last trading day's top 100 High Volume Stocks",
width=800,
height=400,
histnorm="probability"
)
fig.show()
```
### Candlestick
Generates OHLC candlestick charts with optional technical studies and overlays.
**Returns:** Single figure or array of figures (depending on studies)
**Required arguments:**
* `df` - DataFrame with OHLC data (minimum 20 bars)
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - chart labels with HTML support
* `width`, `height` - dimensions (min 700px width, 400px height)
* `open`, `high`, `low`, `close`, `timestamp` - column names
* `entry_px`, `entry_time`, `close_px`, `close_time`, `entry_side` - trade annotations
* `overlay_df`, `overlay_col`, `overlay_col2` - secondary axis data
**Supported Studies (Single Figure):**
`BBANDS`, `DEMA`, `EMA`, `HT_TRENDLINE`, `KAMA`, `MA`, `MAMA`, `MIDPOINT`, `MIDPRICE`, `SAR`, `SAREXT`, `SMA`, `T3`, `TEMA`, `TRIMA`, `WMA`
**Supported Studies (Array of Figures):**
`ADX`, `ADXR`, `APO`, `AROON`, `AROONOSC`, `BOP`, `CCI`, `CMO`, `DX`, `MACD`, `MACDEXT`, `MFI`, `MINUS_DI`, `MINUS_DM`, `MOM`, `PLUS_DI`, `PLUS_DM`, `PPO`, `ROC`, `ROCP`, `ROCR`, `ROCR100`, `RSI`, `STOCH`, `STOCHF`, `STOCHRSI`, `TRIX`, `ULTOSC`, `WILLR`, `ATR`, `MACDFIX`, `NATR`, `TRANGE`
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
from datetime import datetime, timedelta
back_to = (datetime.now() + timedelta(days=-360)).strftime('%Y-%m-%d 23:59:59')
symbol = "SPY"
res = liberator.query(name="daily_bars", symbols=symbol, back_to=back_to)
df_bars = liberator.get_dataframe(res)
res = liberator.query(name="eia_gas_prices", back_to=back_to)
gaspx = liberator.get_dataframe(res)
fig = charting.Candlestick(
df=df_bars,
overlay_df=gaspx,
overlay_col="retail_gas_price",
height=700,
width=800,
xlabel="Date",
title="SPY vs Retail Gas Prices"
)
fig.show()
```
### LineChart
Creates line charts with support for studies and secondary Y-axis.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `cols` - list of column names to plot
* `x_column` - column name for X-axis (defaults to index)
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - chart labels with HTML support
* `width`, `height` - dimensions
* `y2axis_name` - activates secondary Y-axis
* `overlay_df`, `overlay_cols` - secondary axis data
* `study` - technical study name
* `study_columns` - columns for study calculation
* `timeperiod` - study periods
**Supported Studies:**
`BBANDS`, `MAMA`, `DEMA`, `EMA`, `HT_TRENDLINE`, `KAMA`, `MA`, `MIDPOINT`, `MIDPRICE`, `SAR`, `SAREXT`, `SMA`, `T3`, `TEMA`, `TRIMA`, `WMA`
**Example:**
```python theme={null}
import cloudquantcharting
import liberator
res = liberator.query(
symbols='AMZN',
name='Twitter',
back_to='12/22/2020 08:00:00',
as_of='12/22/2020 16:00:00'
)
twitter_df = liberator.get_dataframe(res)
fig = cloudquantcharting.LineChart(
df=twitter_df,
cols=['s-score', 's-buzz', 's', 's-mean', 'sv-score', 's-dispersion', 's-buzz', 's-delta'],
x_column="timestamp"
)
fig.show()
```
### BarChart
Generates bar charts, including stacked bars for repeated X values.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `xcol` - column name for X-axis
* `ycol` - numeric column for Y-axis
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - labels with HTML support
* `width`, `height` - dimensions
* `singlecolor` - boolean for uniform color (defaults `True`)
* `orientation` - `'v'` (vertical) or `'h'` (horizontal)
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
res = liberator.query(name="daily_bars")
df_bars = liberator.get_dataframe(res)
df_bars = df_bars.sort_values(by=['volume'], ascending=[False])
fig = charting.BarChart(
df=df_bars[0:20],
xcol="symbol",
ycol="spread",
title="Spread Price of High Volume Stocks",
width=800,
height=400
)
fig.show()
```
### PieChart
Generates pie/donut charts with customizable hole size.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `labelcol` - column name for labels
* `valuecol` - numeric column for slice size
**Optional arguments:**
* `title` - chart title with HTML support
* `width`, `height` - dimensions
* `hole` - donut hole size (0-1, where 0 = no hole)
* `colors` - color sequence: default CQ palette, or `'reds'`, `'greens'`, `'blues'`
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
values = [
['Jan', '150000', 'USD'],
['Feb', '200000', 'USD'],
['Mar', '330000', 'USD'],
['Apr', '440000', 'USD'],
['May', '555000', 'USD'],
['June', '355000', 'USD'],
['July', '150000', 'USD'],
['Aug', '130000', 'USD'],
['Sept', '330000', 'USD'],
['Oct', '440000', 'USD'],
['Nov', '130000', 'USD'],
['Dec', '330000', 'USD']
]
df = pd.DataFrame(values, columns=["month", "sales", "currency"])
title = "Sales by month"
fig = charting.PieChart(
df=df,
labelcol='month',
valuecol="sales",
title=title,
height=700
)
fig.show()
```
### GroupedBarChart
Generates grouped bar charts comparing multiple numeric columns.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `groups` - column name for grouping
* `values` - list of numeric column names
**Optional arguments:**
* `title`, `group_label`, `value_label` - labels with HTML support
* `width`, `height` - dimensions
* `colors` - color list (cycles through bars)
* `orientation` - `'v'` or `'h'`
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
dataset_name = 'reddit_wallstreetbets_comments'
df = liberator.get_dataframe(liberator.query(name=dataset_name))
df = df.loc[df['vader_body_sentiment_compound'] > 0]
df = df.sort_values(by=['vader_body_sentiment_compound'], ascending=[False])
charting.GroupedBarChart(
df=df[0:5],
groups="symbol",
values=["vader_body_sentiment_pos", "vader_body_sentiment_compound", "textblob_body_sentiment_subjectivity"],
title="WS Bets Sentiment",
group_label="Symbol",
value_label="sentiment score"
)
```
### ScatterPlot
Generates scatter plots with optional bubble sizing and color coding.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `x_column` - numeric column for horizontal axis
* `y_column` - numeric column(s) for vertical axis
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - labels with HTML support
* `width`, `height` - dimensions
* `size_column` - numeric column for bubble size (negative values show red/green)
* `size_multiplier` - multiplier to adjust bubble sizes
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
dataset_name = 'reddit_wallstreetbets_comments'
df = liberator.get_dataframe(liberator.query(name=dataset_name))
df = df.loc[df['vader_body_sentiment_compound'] > 0]
df = df.sort_values(by=['vader_body_sentiment_compound'], ascending=[False])
fig = charting.ScatterPlot(
df=df[0:200],
x_column="textblob_body_sentiment_subjectivity",
y_column=['vader_body_sentiment_neu'],
title="WS Bets Sentiment",
size_multiplier=.15,
width=800,
height=800
)
fig.show()
```
### addNotes
Adds annotations/callouts to existing charts for highlighting specific data points.
**Returns:** `plotly.graph_objs._figure.Figure`
**Arguments:**
* `thefig` - figure from a CloudQuant charting function
* `notes` - list of dictionaries with `x`, `y`, and `note` keys
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
from datetime import datetime, timedelta
back_to = (datetime.now() + timedelta(days=-2)).strftime('%Y-%m-%d 23:59:59')
symbol = "AAPL"
res = liberator.query(name="minute_bars", symbols=symbol, back_to=back_to)
df_bars = liberator.get_dataframe(res)
fig = charting.Candlestick(df=df_bars, title=symbol)
mydata = []
data = {'x': df_bars.timestamp[7], 'y': df_bars.open[7], 'note': "T1"}
mydata.append(data)
data = {'x': df_bars.timestamp[155], 'y': df_bars.open[155], 'note': "T2"}
mydata.append(data)
data = {'x': df_bars.timestamp[1000], 'y': df_bars.open[1000], 'note': "T3"}
mydata.append(data)
fig = charting.addNotes(fig, mydata)
fig.show()
```
# Python SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/python/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator Python API from your own environment.
# Python SDK getting started
The CloudQuant Data Liberator Python SDK provides the simplest way to query and work with CloudQuant datasets from any Python environment.
## Prerequisites
* Python 3
## Installation
Install the third-party dependencies Liberator needs. The `liberator.py` client itself is not on PyPI — download it from your Liberator Profile page (see [Required security files](#required-security-files) below).
```bash Conda theme={null}
conda create -n liberator python=3.7
conda activate liberator
conda install -c anaconda requests pyopenssl cryptography six pytz
conda install pandas
conda install -c conda-forge httpx async_generator pyarrow=8.0.0
```
```bash pip theme={null}
python -m venv liberator
source liberator/bin/activate # Windows: liberator\Scripts\activate
pip install requests pandas httpx async_generator "pyarrow==8.0.0" pyOpenSSL cryptography six pytz
```
## Required security files
You need the following files from your Downloads ZIP file:
* **`liberator.py`** — The CloudQuant Data Liberator API for Python
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
Place `liberator.py` in the same folder as your code or in your Python `site-packages` directory. Place `liberator.json` (and `liberator.pfx` if needed) in your **working directory**, or set explicit paths before querying:
```python theme={null}
import liberator
liberator.auth = '/path/to/liberator.json'
liberator.pfx = '/path/to/liberator.pfx' # releases prior to 2.0 only
```
`liberator.json` is resolved from your working directory unless you set `liberator.auth`.
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
When using Jupyter Notebooks, set the working directory to the location containing `liberator.json` (and `liberator.pfx` if you are on a release prior to 2.0). See [this reference](https://kegui.medium.com/change-jupyter-notebook-startup-folder-windows-608dfcfdc104) for changing the Jupyter Notebook startup folder on Windows.
## Optional: CloudQuant charting
If using CloudQuant Charting (included with the API), install these additional packages:
```bash Conda theme={null}
conda install plotly numpy
conda install -c conda-forge ta-lib
```
```bash pip theme={null}
pip install plotly numpy TA-Lib
```
`TA-Lib` on pip requires the [ta-lib C library](https://github.com/TA-Lib/ta-lib-python) to be installed first. Conda installs both the library and Python bindings together.
## Quick start
```python theme={null}
import liberator
df = liberator.get_dataframe(
liberator.query(
name='daily_bars',
as_of='2020-11-15',
symbols=['AAPL']
)
)
print(df)
```
# R SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/r-language/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator R API.
# R SDK getting started
The CloudQuant Data Liberator R package provides functions for querying datasets and converting results to R DataFrames.
## Required security files
Place the following files from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Quick start
### Query and get DataFrame
```r theme={null}
res = liberator::get_dataframe(liberator::query(
symbols="AAPL",
name="daily_bars",
as_of="2025-01-01",
back_to="2024-12-01"
))
print(res)
```
### Function reference
The `get_dataframe` function accepts a query result and converts it into a DataFrame:
```r theme={null}
get_dataframe <- function(res, muts_order_batches=TRUE)
```
You can send debug output to stderr:
```r theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
symbols="AAPL",
as_of="2019-09-15",
back_to="2019-09-10",
debug_stream=stderr()
)))
```
# RESTful API Getting Started
Source: https://knowledge.cloudquant.com/api-reference/restful/getting-started
Complete guide to querying the CloudQuant Data Liberator RESTful API using curl and standard HTTP tools.
# RESTful API getting started
The CloudQuant Data Liberator API provides RESTful endpoints for querying financial market data. The primary endpoint is `/liberator/query`, which accepts JSON payloads and returns data as JSON objects or Apache Arrow batches.
## Requirements
* `curl`
* `jq`
* `base64`
* Python with `pyarrow` (for Arrow format decoding)
* `pigz` (for compressed Arrow format)
## Authentication
All requests require user credentials and your Liberator base URL (both provided on your Profile page):
```bash theme={null}
URL=
USER=TODO-ADD-YOUR-USERID-HERE
TOKEN=TODO-ADD-YOUR-TOKEN-HERE
```
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `--cert-type P12 --cert liberator.pfx` flags shown in the examples below are only needed for releases prior to 2.0.
## Quick start: JSON queries
### Current value query
```bash theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":["AAPL","IBM"],"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","debug_stream":false}' \
$URL/liberator/query
```
### Time range query
```bash theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","back_to":"2022-12-01","debug_stream":false}' \
$URL/liberator/query
```
### Point-in-time query
```bash theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","as_of":"2022-12-15","debug_stream":false}' \
$URL/liberator/query
```
## Advanced: Apache Arrow format
### Without compression
```bash theme={null}
BATCHES=$(curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":false,"compress":false,"symbols":["AAPL"],"back_to":"2023-01-01","debug_stream":false,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query | jq -r '.[] | select(type=="object") | .batch')
for BATCH in $BATCHES; do
echo $BATCH | base64 -d | python -c "import sys; import pyarrow as pa; print(pa.ipc.open_stream(sys.stdin.buffer.read()).read_next_batch().to_pylist())"
done
```
### With compression
```bash theme={null}
BATCHES=$(curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":false,"compress":true,"symbols":["AAPL"],"back_to":"2023-01-01","debug_stream":false,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query | jq -r '.[] | select(type=="object") | .batch')
for BATCH in $BATCHES; do
echo $BATCH | base64 -d | pigz -dz | python -c "import sys; import pyarrow as pa; print(pa.ipc.open_stream(sys.stdin.buffer.read()).read_next_batch().to_pylist())"
done
```
## Response format
Standard JSON response includes fields such as: `_seq`, `_dsname`, `timestamp`, `symbol`, `open`, `high`, `low`, `close`, `volume`, `vwap`, `spread`, `bidvol`, `askvol`, and `count`.
# Data Catalog
Source: https://knowledge.cloudquant.com/data-catalog/overview
Browse 70+ integrated financial, alternative, and economic datasets available through CloudQuant Data Liberator
# Data catalog - data integrations
CloudQuant Data Liberator integrates with over 70 data providers, offering a comprehensive catalog of financial, alternative, and economic datasets.
## Market & financial data
| Dataset | Description |
| -------------------------------- | ------------------------------- |
| **SpiderRock Options & Futures** | Options and futures market data |
| **CryptoQuote** | Cryptocurrency market data |
| **S\&P Dow Jones Indices** | Index data and analytics |
| **Canari Options** | Options analytics and data |
## Alternative data
| Dataset | Description |
| --------------------------------------- | ------------------------------------ |
| **Alexandria** | Earnings and news sentiment analysis |
| **BMLL Technologies** | Level 3 order book data |
| **Exegy Hidden Order Flow** | Hidden and dark pool order flow |
| **GWI Audience Insight** | Consumer audience analytics |
| **Likefolio Twitter** | Social media sentiment from Twitter |
| **Lunarcrush Crypto Sentiment** | Cryptocurrency social sentiment |
| **Smart Insider (Buybacks)** | Corporate buyback intelligence |
| **New Constructs (Earnings Anomalies)** | Earnings quality analytics |
| **Precision Alpha** | Price prediction signals |
| **Danel SmartScore** | Multi-factor scoring |
## Economic data
| Dataset | Description |
| ------------------------------------------ | ----------------------------- |
| **FRED Release Series** | Federal Reserve Economic Data |
| **Bureau of Economic Analysis GDP** | U.S. GDP data |
| **Bureau of Labor Statistics** | Employment and labor data |
| **U.S. Energy Information Administration** | Energy market data |
| **EuroStat** | European economic statistics |
| **CFTC Commitment of Traders** | Futures positioning data |
## Regulatory & reference data
| Dataset | Description |
| ------------------------------ | --------------------------- |
| **FINRA** | Financial regulatory data |
| **SEC Filing Data** | SEC filings and disclosures |
| **Regalytics Regulatory Data** | Regulatory intelligence |
## Environmental & weather data
| Dataset | Description |
| -------------------- | ----------------------------- |
| **Athenium Weather** | Weather analytics for trading |
| **CustomWeather** | Custom weather data feeds |
| **Ilika Weather** | Weather data and forecasts |
## Additional datasets
| Dataset | Description |
| ---------------------- | ----------------------------------------- |
| **Linkup Jobs** | Job market data |
| **Shipfix Shipping** | Global shipping and freight data |
| **ESG Data Providers** | Environmental, social, governance metrics |
Dataset availability depends on your subscription and entitlements. Use `liberator.datasets()` in Python to see which datasets you have access to. In Liberator **2.4** and later, catalog cards (and `liberator.datasets(schema=True, details=True)`) show date range, symbol count, status, and last updated. Contact CloudQuant to request access to additional datasets.
## Accessing datasets
```python theme={null}
import liberator
# List all datasets you have access to
datasets = liberator.datasets()
print(datasets)
# Query a specific dataset
df = liberator.query(
dataset="your_dataset_name",
symbols=["AAPL"],
as_of="2024-01-01",
back_to="2023-01-01"
)
```
See the [Python Guide](/python-guide/dataset-access) for more details on discovering and querying datasets.
# Aligning Two Datasets Into One
Source: https://knowledge.cloudquant.com/data-science-recipes/aligning-datasets
Strategies for merging two different time series datasets, handling timestamp misalignment, symbol mismatches, and frequency differences.
# Aligning two datasets into one
Merging two different time series datasets into one can be like navigating a minefield — it is one of the trickiest challenges in data science.
## Key challenges
When aligning datasets, you need to consider several critical factors:
1. **Timestamp Availability** — Even if your datasets are timestamped as Daily, you still need to know when the data was available to ensure it can line up.
2. **Symbol Consistency** — Do both datasets contain identical symbols? How do you handle mismatches?
3. **Timeframe Misalignment** — What happens when one dataset operates at 1-minute intervals and another at 5-minute intervals?
4. **Data Expansion Strategy** — When expanding lower-frequency data, should you use first values, last values, or an alternative approach?
5. **Data Aggregation** — When merging in the opposite direction, how do you summarize higher-frequency bars? The choice of average, max, min, or other metrics depends on column content.
## Recommended solution: reindex()
The pandas `reindex()` method allows you to forward fill, back fill, use nearest, or fill with None.
```python theme={null}
df.reindex()
```
For detailed usage, see the [pandas DataFrame.reindex documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reindex.html).
## Alternative: SuperQuery
For datasets within CloudQuant Data Liberator, you can use the **SuperQuery** command to have the system perform the merge automatically. SuperQuery resamples multiple datasets onto a common time axis without manual alignment.
If both of your datasets are available in CloudQuant Data Liberator, SuperQuery is often the simplest approach — it handles the alignment for you. See the [SuperQuery recipe](/data-science-recipes/superquery) for details.
# Extracting Parts of a Date or Time from a Timestamp
Source: https://knowledge.cloudquant.com/data-science-recipes/date-time-extraction
Efficiently extract date and time components from timestamp strings in Python DataFrames using string slicing and pandas methods.
# Extracting parts of a date or time from a timestamp
When working with timestamped market data, you often need to extract specific components like the date, time, or hour. Rather than using slower row-by-row loops, you can use vectorized operations for much better performance.
## String slicing with lambda functions
The primary approach uses `apply()` with lambda functions to extract substrings from timestamp data:
```python theme={null}
df = liberator.get_dataframe(
liberator.query(
name='minute_bars',
symbols='SPY',
as_of='2024-07-26',
back_to='2024-07-25'
)
)
df['Date'] = df.apply(lambda row: str(row.timestamp)[0:10], axis=1)
df['Time'] = df.apply(lambda row: str(row.timestamp)[11:16], axis=1)
df['Hour'] = df.apply(lambda row: str(row.timestamp)[11:13], axis=1)
df['DateTime'] = df.apply(lambda row: str(row.timestamp)[0:16], axis=1)
df['DateHour'] = df.apply(lambda row: str(row.timestamp)[0:13], axis=1)
df[['timestamp', 'Date', 'Time', 'Hour', 'DateTime', 'DateHour']]
```
This creates new columns by slicing the timestamp string at specific positions:
| Column | Slice | Example Output |
| ---------- | --------- | ------------------ |
| `Date` | `[0:10]` | `2024-07-26` |
| `Time` | `[11:16]` | `14:30` |
| `Hour` | `[11:13]` | `14` |
| `DateTime` | `[0:16]` | `2024-07-26 14:30` |
| `DateHour` | `[0:13]` | `2024-07-26 14` |
## Alternative: FLOOR method using Pandas DateTime
You can also use pandas built-in datetime methods:
```python theme={null}
df['seconds'] = df['timestamp'].dt.floor('s')
df['Date'] = pd.to_datetime(df["timestamp"]).dt.date
```
String slicing with lambda functions provides fast processing for extracting timestamp components, making it preferable to manual loops for DataFrame operations.
# Downloading Very Large Datasets
Source: https://knowledge.cloudquant.com/data-science-recipes/large-datasets
Chunk large time series queries into smaller segments to handle network instability and avoid timeouts when downloading market data.
# Downloading very large datasets
For large time series datasets, if you experience network instability, chunk large queries into smaller time segments rather than attempting single monolithic downloads.
## Core strategy
The recommended approach divides large queries into smaller, manageable time-based chunks (typically 30-minute segments). This methodology:
* Reduces network timeout and data corruption likelihood
* Automatically retries failed chunks without losing progress
* Continues downloading even when individual chunks fail
* Provides clear feedback on download progress
## Complete Python implementation
```python theme={null}
import liberator
import pandas as pd
from datetime import datetime, timedelta
from typing import Union, List, Optional
import time
def getdata(dataset: str,
start_date: Union[str, datetime],
end_date: Union[str, datetime],
symbols: Union[str, List[str]],
chunk_minutes: int = 30,
max_retries: int = 3,
retry_delay: float = 1.0,
verbose: bool = True) -> pd.DataFrame:
"""
Download data by breaking the time range into smaller chunks.
Args:
dataset: Dataset name (e.g., 'spiderrock_printsets_indexed_single_tenant')
start_date: Start date/time as string 'YYYY-MM-DD [HH:MM:SS]' or datetime
end_date: End date/time as string 'YYYY-MM-DD [HH:MM:SS]' or datetime
symbols: Single symbol string or list of symbols
chunk_minutes: Minutes per chunk (default 30)
max_retries: Maximum retries per chunk (default 3)
retry_delay: Delay between retries in seconds (default 1.0)
verbose: Print progress messages (default True)
Returns:
pandas.DataFrame: Combined data for the entire time range
"""
# Parse and validate inputs
start_dt = _parse_datetime(start_date)
end_dt = _parse_datetime(end_date)
if isinstance(symbols, str):
symbols = [symbols]
if verbose:
print(f"Downloading {dataset} from {start_dt} to {end_dt} for {len(symbols)} symbols")
print(f"Using {chunk_minutes}-minute chunks")
# Generate time chunks
chunks = _generate_time_chunks(start_dt, end_dt, chunk_minutes)
if verbose:
print(f"Processing {len(chunks)} time chunks...")
# Download each chunk
all_dfs = []
failed_chunks = []
empty_chunks = []
for i, (chunk_start, chunk_end) in enumerate(chunks):
if verbose:
end_display = chunk_end.strftime('%H:%M:%S') if chunk_end.second != 0 else chunk_end.strftime('%H:%M')
print(f" Processing chunk {i+1}/{len(chunks)} ({chunk_start.strftime('%H:%M')} - {end_display})", end="")
# Download chunk with retries
result = _download_chunk_with_retry(
dataset=dataset,
start_time=chunk_start,
end_time=chunk_end,
symbols=symbols,
max_retries=max_retries,
retry_delay=retry_delay,
verbose=verbose
)
if result is None:
# Actual failure (exception occurred)
failed_chunks.append((chunk_start, chunk_end))
if verbose:
print(" - FAILED")
elif len(result) == 0:
# Empty result (no data for this time period)
empty_chunks.append((chunk_start, chunk_end))
if verbose:
print(" - EMPTY")
else:
# Success with data
all_dfs.append(result)
if verbose:
print(f" - SUCCESS ({len(result):,} rows)")
# Report results
if verbose:
total_rows = sum(len(df) for df in all_dfs)
print(f"Complete: {len(all_dfs)} chunks with data, {len(empty_chunks)} empty chunks, {len(failed_chunks)} failed chunks")
print(f"Total rows downloaded: {total_rows:,}")
if len(failed_chunks) > 0:
print(f"Warning: {len(failed_chunks)} chunks failed due to errors")
# Combine all dataframes
if all_dfs:
combined_df = pd.concat(all_dfs, ignore_index=True)
# Sort by time if columns exist
if 'muts' in combined_df.columns and '_seq' in combined_df.columns:
combined_df = combined_df.sort_values(['muts', '_seq'])
combined_df.reset_index(drop=True, inplace=True)
elif 'timestamp' in combined_df.columns:
combined_df = combined_df.sort_values('timestamp')
combined_df.reset_index(drop=True, inplace=True)
return combined_df
else:
print("No data downloaded successfully")
return pd.DataFrame()
def _parse_datetime(dt_input: Union[str, datetime]) -> datetime:
"""Parse string or datetime input into datetime object."""
if isinstance(dt_input, datetime):
return dt_input
dt_str = str(dt_input).strip()
# Try different datetime formats
formats = [
'%Y-%m-%d %H:%M:%S.%f', # Full datetime with microseconds
'%Y-%m-%d %H:%M:%S', # Full datetime
'%Y-%m-%d %H:%M', # Date with hour:minute
'%Y-%m-%d', # Date only
]
for fmt in formats:
try:
return datetime.strptime(dt_str, fmt)
except ValueError:
continue
raise ValueError(f"Unable to parse datetime: {dt_input}")
def _generate_time_chunks(start_dt: datetime, end_dt: datetime, chunk_minutes: int) -> List[tuple]:
"""Generate list of (start, end) datetime tuples for chunks."""
chunks = []
current_start = start_dt
chunk_delta = timedelta(minutes=chunk_minutes)
while current_start < end_dt:
current_end = min(current_start + chunk_delta, end_dt)
chunks.append((current_start, current_end))
current_start = current_end
return chunks
def _download_chunk_with_retry(dataset: str,
start_time: datetime,
end_time: datetime,
symbols: List[str],
max_retries: int,
retry_delay: float,
verbose: bool = True) -> Optional[pd.DataFrame]:
"""Download a single time chunk with retry logic."""
# Format times as strings for liberator
start_str = start_time.strftime('%Y-%m-%d %H:%M:%S')
end_str = end_time.strftime('%Y-%m-%d %H:%M:%S')
for attempt in range(max_retries):
try:
# Query the data
query_result = liberator.query(
name=dataset,
symbols=symbols,
back_to=start_str,
as_of=end_str
)
# Convert to dataframe
df = liberator.get_dataframe(query_result)
# Return the dataframe (could be empty, but that's not a failure)
return df
except Exception as e:
if attempt < max_retries - 1: # Don't sleep on last attempt
time.sleep(retry_delay)
else:
if verbose:
print(f" Failed chunk {start_time.strftime('%H:%M')}-{end_time.strftime('%H:%M')} after {max_retries} attempts: {str(e)}")
return None # Return None only on actual exception
return None
# Example usage functions
def download_single_day(dataset: str, date: str, symbols: Union[str, List[str]], **kwargs) -> pd.DataFrame:
"""
Convenience function to download a single day's data.
Args:
dataset: Dataset name
date: Date as 'YYYY-MM-DD'
symbols: Symbol(s) to download
**kwargs: Additional arguments passed to getdata()
Returns:
pandas.DataFrame: Day's data
"""
start_date = f"{date} 00:00:00"
end_date = f"{date} 23:59:59"
return getdata(
dataset=dataset,
start_date=start_date,
end_date=end_date,
symbols=symbols,
**kwargs
)
def download_date_range(dataset: str,
start_date: str,
end_date: str,
symbols: Union[str, List[str]],
**kwargs) -> pd.DataFrame:
"""
Download data across multiple days.
Args:
dataset: Dataset name
start_date: Start date as 'YYYY-MM-DD'
end_date: End date as 'YYYY-MM-DD'
symbols: Symbol(s) to download
**kwargs: Additional arguments passed to getdata()
Returns:
pandas.DataFrame: Multi-day data
"""
start_dt = _parse_datetime(f"{start_date} 00:00:00")
end_dt = _parse_datetime(f"{end_date} 23:59:59.999")
return getdata(
dataset=dataset,
start_date=start_dt,
end_date=end_dt,
symbols=symbols,
**kwargs
)
```
## Usage examples
### Single day, multiple symbols
```python theme={null}
df1 = download_single_day(
dataset='spiderrock_printsets_indexed_single_tenant',
date='2025-09-08',
symbols=['AAPL', 'GOOGL', 'MSFT']
)
```
### Custom time range
```python theme={null}
df2 = getdata(
dataset='spiderrock_printsets_indexed_single_tenant',
start_date='2025-09-08 09:30:00',
end_date='2025-09-08 16:00:00',
symbols='AAPL',
chunk_minutes=15,
verbose=False
)
```
### Multi-day download
```python theme={null}
df3 = download_date_range(
dataset='spiderrock_printsets_indexed_single_tenant',
start_date='2025-09-03',
end_date='2025-09-06',
symbols=['AAPL', 'GOOGL'],
chunk_minutes=120 # 2-hour chunks for longer periods
)
```
## Sample output
```
Downloading spiderrock_printsets_indexed_single_tenant from 2025-09-03 00:00:00
to 2025-09-06 23:59:59.999000 for 2 symbols
Using 120-minute chunks
Processing 48 time chunks...
Processing chunk 1/48 (00:00 - 02:00) - EMPTY
Processing chunk 2/48 (02:00 - 04:00) - EMPTY
Processing chunk 3/48 (04:00 - 06:00) - EMPTY
Processing chunk 4/48 (06:00 - 08:00) - EMPTY
Processing chunk 5/48 (08:00 - 10:00) - SUCCESS (143,762 rows)
Processing chunk 6/48 (10:00 - 12:00) - SUCCESS (168,790 rows)
...
Complete: 13 chunks with data, 35 empty chunks, 0 failed chunks
Total rows downloaded: 959,576
```
For multi-day downloads, increase `chunk_minutes` to reduce the number of API calls. A value of 120 (2 hours) works well for longer date ranges.
The chunking approach is designed for network resilience. If you are on a stable connection and downloading moderately sized datasets, a single query may be simpler and faster.
# Using Market Calendar (mcal) to Identify Trading Dates
Source: https://knowledge.cloudquant.com/data-science-recipes/market-calendar
Programmatically determine trading dates using the pandas_market_calendars library for NYSE, CME, CBOE, NASDAQ, and other exchanges.
# Using Market Calendar (mcal) to identify trading dates
When retrieving market data, it is beneficial to programmatically determine which dates represent actual trading dates. The **Market Calendar (mcal)** library enables this functionality.
## US stock market calendars
Market Calendar supports multiple US exchanges including NYSE, CME, CBOE, and NASDAQ.
## Implementation example
```python theme={null}
# Get NYSE Schedule for date range
# Documentation: https://pandas-market-calendars.readthedocs.io/en/latest/usage.html
start_date = '2024-01-01'
end_date = '2024-06-01'
import pandas as pd
import pandas_market_calendars as mcal
nyse = mcal.get_calendar('NYSE')
schedule = nyse.schedule(start_date=start_date, end_date=end_date)
schedule
schedule.market_open.iloc[0] # open time first date
schedule.market_close.iloc[0] # close time first date
schedule.market_open.iloc[-1] # open time last date
schedule.market_close.iloc[-1] # close time last date
schedlist = list(schedule.index)
schedlist
```
Although mcal is generally extremely reliable, it is an external library and has on occasion missed a market holiday. Implement defensive coding practices and verify results independently.
The library demonstrates strong reliability for US market holidays but cannot be vouched for with equal confidence regarding international markets.
## Resources
* [Official pandas\_market\_calendars Documentation](https://pandas-market-calendars.readthedocs.io/en/latest/usage.html)
# Melt and Wide to Long - Unpivoting a Pivot
Source: https://knowledge.cloudquant.com/data-science-recipes/melt-and-unpivot
Convert DataFrames from wide to long format using pandas melt and wide_to_long methods.
# Melt and wide to long - unpivoting a pivot
Pandas provides two methods for converting DataFrames from wide to long format (unpivoting).
## Melt method
Unpivot a DataFrame from wide to long format, optionally leaving identifiers set.
```python theme={null}
df.melt()
```
The `melt` function provides flexibility when restructuring data from a wide format into a long format. You can specify which columns serve as identifiers and which become variable-value pairs.
Use `melt` when you need fine-grained control over which columns become identifiers and which become variable-value pairs.
**Documentation:** [pandas.melt](https://pandas.pydata.org/docs/reference/api/pandas.melt.html)
## Wide to long method
Unpivot a DataFrame from wide to long format. Less flexible but more user-friendly than melt.
```python theme={null}
pd.wide_to_long()
```
The `wide_to_long` function offers a more accessible alternative to melt, trading some flexibility for improved usability when performing standard unpivoting operations.
Use `wide_to_long` for simpler, standard unpivoting operations where ease of use is more important than flexibility.
**Documentation:** [pandas.wide\_to\_long](https://pandas.pydata.org/docs/reference/api/pandas.wide_to_long.html)
# A Merge or Join
Source: https://knowledge.cloudquant.com/data-science-recipes/merge-and-join
Combine two DataFrames by columns using pandas merge, join, and merge_asof operations.
# A merge or join
A merge or join operation expands the number of **columns** in a resulting DataFrame by combining data from two tables based on shared values. This differs from concatenation, which combines **rows**.
## Merge vs. concat
| Operation | What it does |
| -------------- | -------------------------------------------------------------------------------------------------------------------- |
| **CONCAT** | Combines rows from two DataFrames (e.g., two 10-row DataFrames yield 20 rows). Generally requires identical columns. |
| **MERGE/JOIN** | Combines columns by matching rows on a shared column, typically increasing column count. |
## pd.merge()
Merging requires a **left** dataset, a **right** dataset, and a common column specified with the `on` parameter.
### Join types
The `how` parameter controls which rows appear in results:
| Join Type | Behavior |
| --------- | ------------------------------------------------------------------------ |
| `inner` | **(DEFAULT)** Only rows where the merge column exists in both DataFrames |
| `left` | All rows from the left DataFrame; `NaN` where right data is missing |
| `right` | All rows from the right DataFrame; `NaN` where left data is missing |
| `outer` | All rows from both DataFrames; `NaN` where matches do not exist |
Use `indicator=True` to add a column showing the merge source (`'both'`, `'left_only'`, `'right_only'`) — especially useful when debugging outer joins.
### Basic examples
```python theme={null}
# Standard merge on a shared column
res = pd.merge(left, right, on='shared_column')
# Left join
res = pd.merge(left, right, on='shared_column', how='left')
# Merge with different column names
res = pd.merge(left, right, left_on='leftcol2',
right_on='rightcol7', how='left')
# Select specific columns from right before merging
res = pd.merge(left, right[['col1', 'col2', 'col3']],
on='shared_column')
# Check overlap before merging
left['datesym'].isin(right['datesym']).value_counts()
```
## merge\_asof()
`merge_asof` offers additional flexibility for time-based merging:
* Merging on multiple values
* Specifying tolerance thresholds
### Example: matching quotes to trades
```python theme={null}
pd.merge_asof(trades, quotes, on='time', by='ticker')
# With tolerance
pd.merge_asof(trades, quotes, on='time', by='ticker',
tolerance=pd.Timedelta('2ms'))
```
If `merge_asof()` fails, try removing the `tolerance` parameter.
## Reference
* [pandas.merge\_asof documentation](https://pandas.pydata.org/docs/reference/api/pandas.merge_asof.html)
# Summarizing Data with a Pivot Table
Source: https://knowledge.cloudquant.com/data-science-recipes/pivot-tables
Use pandas pivot tables to summarize, reorganize, and explore large market datasets by aggregating data into meaningful categories.
# Summarizing data with a pivot table
A pivot table is a data analysis tool that enables you to summarize, reorganize, and explore large datasets. It aggregates data, rotates perspectives, and filters, sorts, and groups information into meaningful categories.
## Key benefits
* **Simplify Complex Data** — Break down large datasets into manageable summaries
* **Identify Trends** — Reveal patterns not immediately obvious in raw data
* **Flexible Analysis** — Quickly change layout and structure to answer different questions
* **Efficiency** — Automate data analysis for faster, regular reporting
## Pandas pivot table syntax
```python theme={null}
import pandas as pd
pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'], aggfunc='sum')
```
## Practical example: analyzing odd lots in AAPL trades
This example demonstrates analyzing AAPL trade data by:
1. Extracting trades between 9:30 AM and 4 PM
2. Creating an `odd_lots` boolean column (shares \< 100)
3. Extracting hour from timestamp
4. Pivoting to count shares by hour and lot size
5. Calculating percentage of odd lots per hour
```python theme={null}
import liberator
%time df = liberator.get_dataframe(
liberator.query(
name='trades',
symbols='AAPL',
as_of='2024-07-30 15:59:58',
back_to='2024-07-30 09:30:01'
)
)
df['odd_lots'] = df['shares'] < 100
df['Hour'] = df.apply(lambda row: str(row.timestamp)[11:13], axis=1)
pivot_table = df.pivot_table(
values='shares',
index='Hour',
columns='odd_lots',
aggfunc='sum',
fill_value=0
)
pivot_table['% Odd Lots'] = (
pivot_table[True] / (pivot_table[True] + pivot_table[False]) * 100
)
pivot_table
```
## Parameter reference
| Parameter | Purpose |
| ------------ | -------------------------------------------------------- |
| `aggfunc` | Specifies aggregation function (`'sum'`, `'mean'`, etc.) |
| `values` | Target column for aggregation |
| `index` | Column(s) defining row groupings |
| `columns` | Column(s) defining column splits |
| `fill_value` | Default value for missing data |
# Python Data Science Shortcuts and Snippets
Source: https://knowledge.cloudquant.com/data-science-recipes/shortcuts-and-snippets
Practical Python solutions for common data science challenges when working with DataFrames and market data.
# Python data science shortcuts and snippets
The solutions to most challenges in data science are best solved by experimentation and failure. This guide focuses on practical, daily-use solutions for common data science tasks.
Large Language Models like ChatGPT can help format data effectively when time is limited. They can be a valuable companion for quick data transformations.
This collection of shortcuts and snippets covers the most frequently encountered data manipulation tasks. While it does not provide comprehensive training on data science fundamentals or Lambda functions, these tools can rapidly transform large DataFrames.
The recipes in this section cover specific topics in depth. Use the navigation to explore individual techniques like date extraction, pivot tables, merging datasets, and more.
# SuperQuery
Source: https://knowledge.cloudquant.com/data-science-recipes/superquery
Resample pandas DataFrames from CloudQuant Data Liberator into a common time axis by querying multiple datasets simultaneously.
# SuperQuery
SuperQuery enables you to resample pandas DataFrames from CloudQuant Data Liberator into a common time axis by querying multiple datasets simultaneously.
## Initial setup
```python theme={null}
import liberator
symbols = ['TSLA']
as_of = '2024-06-07'
back_to = '2024-06-04'
%time df1 = liberator.get_dataframe(
liberator.query(
symbols=symbols,
name='minute_bars',
as_of=as_of,
back_to=back_to
)
)
liberator.get_dataframe(
liberator.query(
name='daily_bars',
symbols=symbols,
as_of=as_of,
back_to=back_to
)
)
```
This loads two independent datasets: minute bars and daily bars.
## Querying multiple datasets together
SuperQuery lets you specify how multiple datasets are reindexed and merged together using the `superq_resample_rule` parameter.
### Daily (1D) resampling
For a three-day query using daily frequency, the result contains three rows with daily bars and the final minute bar of each day:
```python theme={null}
df = liberator.get_dataframe(
liberator.query(
symbols=symbols,
name=['daily_bars', 'minute_bars'],
as_of=as_of,
back_to=back_to,
superq_periods_per_batch_override=5,
superq_resample_rule='1D'
)
)
df
```
### Hourly (60T) resampling
For a three-day query using 60-minute frequency, the result contains 24 rows per day. Daily bar data is timestamped at 8pm:
```python theme={null}
%time df4 = liberator.get_dataframe(
liberator.query(
symbols=symbols,
name=['daily_bars', 'minute_bars'],
as_of=as_of,
back_to=back_to,
superq_periods_per_batch_override=5,
superq_resample_rule='60T'
)
)
df
```
## The superq\_resample\_rule parameter
The `superq_resample_rule` parameter uses pandas "Offset Aliases" to define the resampling frequency.
Common values:
| Rule | Frequency |
| ----- | ---------- |
| `1T` | 1 minute |
| `5T` | 5 minutes |
| `15T` | 15 minutes |
| `60T` | 60 minutes |
| `1D` | 1 day |
For a complete list of offset aliases, see the [pandas time series offset aliases documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases).
# Azure Blob Storage
Source: https://knowledge.cloudquant.com/datasource-config/azure-blob
Configure Azure Blob Storage datasources
# Azure Blob Storage
Azure Blob Storage datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from Azure Blob containers. CloudQuant Data Liberator mounts the container via FUSE.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest on Azure Blob, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | -------------------------- |
| `connection_type` | string | Must be `"azure"` |
| `account` | string | Azure storage account name |
| `key` | string | Storage account access key |
| `container` | string | Blob container name |
### Optional fields
| Field | Type | Default | Description |
| ------------- | ------ | ------- | ------------------------------------------------------------ |
| `endpoint` | string | | Blob endpoint URL (override for sovereign clouds or Azurite) |
| `prefix` | string | `""` | Blob prefix (virtual directory) within the container |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
| `object_key` | string | | Object key pattern for blob selection |
The `endpoint` field is only needed for non-standard Azure environments such as Azure Government, Azure China, or local development with Azurite. For standard Azure public cloud, CloudQuant Data Liberator derives the endpoint from the account name.
### Example connection
```json theme={null}
{
"name": "azure-market-data",
"connection_type": "azure",
"account": "mystorageaccount",
"key": "base64encodedstorageaccountkey==",
"container": "market-data",
"prefix": "equities/daily/"
}
```
Storage account keys grant full access to the entire storage account. Rotate keys regularly and use environment variables or a secrets manager to inject them at deployment time.
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "azure-eod-prices",
"connection_type": "azure",
"account": "caboretlstorage",
"key": "xYzAbCdEfGhIjKlMnOpQrStUvWxYz0123456789==",
"container": "eod-data",
"prefix": "prices/us-equities/"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-eod",
"connection": "azure-eod-prices",
"data_args": {
"file_pattern": "eod_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "date",
"data_dt_format": "%Y-%m-%d",
"data_dt_timezone": "America/New_York",
"data_key_column": "ticker",
"fname_dt_regex": "eod_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "ticker", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "date", "type": "string", "group": "time", "description": "Trade date" },
{ "name": "open", "type": "double", "group": "value", "description": "Open price" },
{ "name": "high", "type": "double", "group": "value", "description": "High price" },
{ "name": "low", "type": "double", "group": "value", "description": "Low price" },
{ "name": "close", "type": "double", "group": "value", "description": "Close price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Daily volume" },
{ "name": "adj_close", "type": "double", "group": "value", "description": "Adjusted close price" }
]
}
```
## Azure setup requirements
Ensure the following before configuring your Azure Blob connection:
1. **Storage account** exists and is accessible from the CloudQuant Data Liberator host
2. **Container** is created within the storage account
3. **Access key** is available (found under Storage Account > Access Keys in the Azure Portal)
4. **Network access** is permitted — if the storage account uses firewall rules, add the CloudQuant Data Liberator host's IP to the allowed list
For initial testing, use Azure Storage Explorer to verify your files are accessible before configuring the CloudQuant Data Liberator connection. This eliminates credential and network issues early.
See the [Azure Blob Storage integration guide](/integrations/azure-blob-storage) for additional setup details.
# CIFS/SMB
Source: https://knowledge.cloudquant.com/datasource-config/cifs
Configure Windows/Samba network share datasources
# CIFS/SMB
CIFS/SMB datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from Windows file shares or Samba servers. This is commonly used in enterprise environments where data is published to shared network drives.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest from CIFS/SMB shares, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ---------- | ------ | ---------------------------------------------------------------------- |
| `username` | string | Windows/Samba domain username |
| `password` | string | Password for authentication |
| `server` | string | Server hostname or IP address |
| `path` | string | UNC path to the share (e.g., `"sharename"` or `"sharename/subfolder"`) |
The `connection_type` for CIFS/SMB connections is inferred from the presence of the `server` and `path` fields. Unlike other connection types, there is no explicit `connection_type` field required.
### Optional fields
| Field | Type | Default | Description |
| -------- | ------ | ------- | -------------------------------------------- |
| `prefix` | string | `""` | Subdirectory within the share to use as root |
### Example connection
```json theme={null}
{
"name": "cifs-shared-drive",
"username": "DOMAIN\\datauser",
"password": "w1nd0wsP@ss",
"server": "fileserver.corp.example.com",
"path": "MarketData",
"prefix": "daily/equities/"
}
```
Use a dedicated service account with read-only permissions on the share. Avoid using personal credentials, as password changes would break the connection.
### Domain authentication
For Active Directory environments, include the domain in the username:
```json theme={null}
{
"username": "CORP\\svc-liberator"
}
```
Or use the UPN format:
```json theme={null}
{
"username": "svc-liberator@corp.example.com"
}
```
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "cifs-risk-reports",
"username": "CORP\\svc-liberator",
"password": "s3rv1ceAcct!",
"server": "nas01.corp.example.com",
"path": "RiskReports",
"prefix": "daily/"
}
```
### Dataset
```json theme={null}
{
"name": "daily-risk-metrics",
"connection": "cifs-risk-reports",
"data_args": {
"file_pattern": "risk_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "report_date",
"data_dt_format": "%Y-%m-%d",
"data_dt_timezone": "America/New_York",
"data_key_column": "portfolio_id",
"fname_dt_regex": "risk_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "portfolio_id", "type": "string", "group": "key", "description": "Portfolio identifier" },
{ "name": "report_date", "type": "string", "group": "time", "description": "Report date" },
{ "name": "var_95", "type": "double", "group": "value", "description": "95th percentile VaR" },
{ "name": "var_99", "type": "double", "group": "value", "description": "99th percentile VaR" },
{ "name": "cvar", "type": "double", "group": "value", "description": "Conditional VaR" },
{ "name": "beta", "type": "double", "group": "value", "description": "Portfolio beta" },
{ "name": "sharpe", "type": "double", "group": "value", "description": "Sharpe ratio" }
]
}
```
## Network requirements
| Requirement | Detail |
| ------------------ | ------------------------------------------------------------------------ |
| **Outbound port** | TCP port 445 (SMB) to the file server |
| **DNS resolution** | The server hostname must resolve from the CloudQuant Data Liberator host |
| **Authentication** | NTLM or Kerberos (depending on domain configuration) |
| **SMB version** | SMB 2.0 or higher recommended; SMBv1 is deprecated and insecure |
If the CloudQuant Data Liberator host is Linux-based, install `cifs-utils` for CIFS mount support.
```bash Debian/Ubuntu theme={null}
apt install cifs-utils
```
```bash RHEL/CentOS theme={null}
yum install cifs-utils
```
## Troubleshooting
| Issue | Possible Cause | Solution |
| ------------------------------------ | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Mount fails with "Permission denied" | Incorrect credentials or insufficient share permissions | Verify credentials and ensure the service account has read access to the share |
| Mount fails with "Host is down" | Network connectivity or firewall blocking port 445 | Check firewall rules and verify SMB port is reachable |
| Files not found | Incorrect `path` or `prefix` | Verify the UNC path using `smbclient` or Windows Explorer |
| Encoding errors | Files use non-UTF-8 encoding | Set `encoding` in data\_args (common alternatives: `"latin-1"`, `"cp1252"` for Windows-origin files) |
# FTPS
Source: https://knowledge.cloudquant.com/datasource-config/ftps
Configure FTP over TLS/SSL datasources
# FTPS
FTPS (FTP over TLS/SSL) datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from FTP servers secured with TLS encryption. This is commonly used with legacy financial data feeds and vendor systems that require explicit FTPS connections.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest over FTPS, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | --------------------------------- |
| `connection_type` | string | Must be `"ftps"` |
| `host` | string | FTP server hostname or IP address |
| `user` | string | Username for authentication |
### Optional fields
| Field | Type | Default | Description |
| ------------- | ------ | ------- | -------------------------------------------------------------------------------- |
| `port` | int | `21` | FTP control port number |
| `password` | string | | Password for authentication |
| `key` | string | | Client certificate private key (PEM format) for certificate-based authentication |
| `no_verify` | bool | `false` | Skip TLS certificate verification |
| `prefix` | string | `""` | Remote directory path to use as root |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
Setting `no_verify` to `true` disables TLS certificate validation. This should only be used for testing or when connecting to servers with self-signed certificates on trusted internal networks. Never use this in production with external servers.
You must provide either `password` or `key` for authentication. Password authentication is most common for FTPS. Client certificate authentication (`key`) is used when the server requires mutual TLS.
### Example connection
```json theme={null}
{
"name": "ftps-vendor-feed",
"connection_type": "ftps",
"host": "ftp.securefeed.example.com",
"port": 21,
"user": "datafeed",
"password": "v3nd0rP@ss",
"prefix": "/outgoing/daily/"
}
```
### Example connection (self-signed certificate)
```json theme={null}
{
"name": "ftps-internal-data",
"connection_type": "ftps",
"host": "ftp.internal.corp.net",
"port": 990,
"user": "etl-service",
"password": "internalPass123",
"no_verify": true,
"prefix": "/data/exports/"
}
```
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "ftps-reference-data",
"connection_type": "ftps",
"host": "ftps.refdata.example.com",
"port": 21,
"user": "cq-download",
"password": "refDataP@ss",
"prefix": "/reference/securities/"
}
```
### Dataset
```json theme={null}
{
"name": "security-master",
"connection": "ftps-reference-data",
"data_args": {
"file_pattern": "secmaster_*.csv",
"sep_override": "|",
"encoding": "utf-8",
"data_dt_column": "effective_date",
"data_dt_format": "%Y-%m-%d",
"data_dt_timezone": "America/New_York",
"data_key_column": "cusip",
"fname_dt_regex": "secmaster_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "cusip", "type": "string", "group": "key", "description": "CUSIP identifier" },
{ "name": "effective_date", "type": "string", "group": "time", "description": "Effective date" },
{ "name": "ticker", "type": "string", "group": "value", "description": "Ticker symbol" },
{ "name": "company_name", "type": "string", "group": "value", "description": "Company name" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Primary exchange" },
{ "name": "sector", "type": "string", "group": "value", "description": "GICS sector" },
{ "name": "shares_outstanding", "type": "int64", "group": "value", "description": "Shares outstanding" }
]
}
```
## FTPS vs SFTP
| Feature | FTPS | SFTP |
| --------------------- | ------------------------------- | -------------- |
| Protocol | FTP + TLS | SSH subsystem |
| Default port | 21 (explicit) or 990 (implicit) | 22 |
| Firewall friendliness | Requires data port range | Single port |
| Common use | Legacy financial feeds | Modern systems |
| Certificate | TLS/SSL certificates | SSH host keys |
If your data vendor supports both FTPS and SFTP, prefer SFTP. It uses a single port, simplifies firewall configuration, and is generally easier to manage.
## Network requirements
| Requirement | Detail |
| ------------------ | ----------------------------------------------------------------------------------- |
| **Outbound port** | TCP port 21 (control) + passive data port range to the FTP server |
| **Passive mode** | FTP passive mode is required; ensure the passive port range is open |
| **DNS resolution** | The hostname must resolve from the CloudQuant Data Liberator host |
| **TLS support** | Server must support explicit TLS (AUTH TLS) on port 21, or implicit TLS on port 990 |
# HubSpot
Source: https://knowledge.cloudquant.com/datasource-config/hubspot
Connect HubSpot CRM objects as Liberator datasets for cross-object analytics
# HubSpot
HubSpot connections let Liberator read CRM objects (contacts, companies, deals, and related associations) and expose them as standard Liberator datasets. Once onboarded, HubSpot data uses the same entitlements, query API, and AI assistant surfaces as file- and database-backed sources.
HubSpot connectivity is available in Liberator **2.3** and later.
## Prerequisites
* A HubSpot account with API access to the objects you intend to query
* A HubSpot private app token (or equivalent API credential) with read scopes for those objects
* **Admin** or **Super Admin** role in Liberator to create connections and datasets
## Connect HubSpot
Go to **Connections → Add Connection** and select **HubSpot**.
Paste your HubSpot API token into the credential field. Liberator stores the secret with the connection and does not display it again after save.
Click **Test Connection**. On success, Liberator can list available HubSpot objects for dataset onboarding.
Create a dataset against the connection, choose the HubSpot object (or association view) to expose, and complete the usual column / timestamp mapping steps.
## What you can query
HubSpot-backed datasets support the same Liberator query parameters as other sources (`symbols` / key fields, `as_of`, `back_to`, column selection, and so on), subject to the object schema HubSpot returns. Use them when you want CRM facts alongside market or file-backed datasets without a separate extract pipeline.
## Tips
* Scope the HubSpot token to **read-only** access for the objects you need.
* Prefer one Liberator dataset per HubSpot object you query often; keep association-heavy views as separate datasets when schemas differ.
* After rotating a HubSpot token, update the connection credentials and re-test before users query.
* In Liberator **2.4** and later, HubSpot queries wait about 10 seconds and retry when the CRM API returns a rate-limit error, so large initial syncs are less likely to fail mid-run.
## Related
Connection + dataset architecture and shared configuration fields
Full Liberator 2.3 release notes, including the HubSpot connector
Rate-limit retries and other 2.4 reliability improvements
# Local File (CSV/TSV)
Source: https://knowledge.cloudquant.com/datasource-config/local-file
Configure local or mounted file datasources for CloudQuant Data Liberator
# Local file (CSV/TSV)
Local file datasources read CSV, TSV, or other delimited flat files from a directory on the CloudQuant Data Liberator server or a mounted filesystem. This is the simplest file-based connection type and serves as the foundation for understanding all other file-based sources.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest on this connection, including Parquet, Arrow, Excel, XML, HDF5, PSV (2.1+), and PCAP (2.2+).
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | ---------------------------------------------------- |
| `connection_type` | string | Must be `"file"` |
| `behavior` | string | Must be `"file"` |
| `location` | string | Absolute path to the directory containing data files |
The `location` field should point to a **directory**, not an individual file. CloudQuant Data Liberator will scan the directory for files matching the `file_pattern` in `data_args`.
### Example connection
```json theme={null}
{
"name": "local-trades-connection",
"connection_type": "file",
"behavior": "file",
"location": "/data/trades"
}
```
## Dataset configuration (data\_args)
All file-based datasources share the same `data_args` fields. These control how CloudQuant Data Liberator finds, parses, and interprets your files.
### Required fields
| Field | Type | Description |
| ----------------- | -------------- | ---------------------------------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern to match files, e.g., `"*.csv"`, `"prefix_*.tsv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format string, or special values: `"muts"`, `"uts"`, `"nuts"`, `"datetime"`, `"date"` |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------------------------------------------------- |
| `sep_override` | string | `","` | Delimiter character: `","` (comma), `"\t"` (tab), `"\|"` (pipe), `";"` (semicolon) |
| `encoding` | string | `"utf-8"` | File encoding (e.g., `"utf-8"`, `"latin-1"`, `"ascii"`) |
| `data_dt_timezone` | string | `"UTC"` | Timezone of source data, e.g., `"UTC"`, `"America/New_York"` |
| `fname_dt_regex` | string | | Regex to extract a date from the filename |
| `fname_dt_format` | string | | strptime format for the date extracted by `fname_dt_regex` |
| `fname_dt_timezone` | string | | Timezone of the filename-derived date |
| `fname_dt_nudge` | int | `0` | Microsecond offset applied to filename-derived dates |
| `fname_dt_approx_seconds` | int | | Approximate number of seconds of data per file (used for query optimization) |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order for the resulting Arrow table |
| `arrow_timestamp` | bool | `true` | Whether to generate the human-readable `timestamp` column |
Set `fname_dt_approx_seconds` to `86400` for daily files. This helps CloudQuant Data Liberator skip files outside the query's time range, significantly improving performance for large directories.
## Complete example
Below is a full configuration showing both the connection and a dataset for daily trade CSV files.
### Connection
```json theme={null}
{
"name": "local-daily-trades",
"connection_type": "file",
"behavior": "file",
"location": "/data/daily-trades"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "local-daily-trades",
"data_args": {
"file_pattern": "trades_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "trade_time",
"data_dt_format": "%Y-%m-%d %H:%M:%S",
"data_dt_timezone": "America/New_York",
"data_key_column": "symbol",
"fname_dt_regex": "trades_(\\d{4}-\\d{2}-\\d{2})\\.csv",
"fname_dt_format": "%Y-%m-%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" }
]
}
```
Ensure the CloudQuant Data Liberator process has read permissions on the `location` directory and all files within it. Permission errors will cause silent failures during query execution.
## Tab-separated files (TSV)
For TSV files, set `sep_override` to `"\t"`:
```json theme={null}
{
"data_args": {
"file_pattern": "*.tsv",
"sep_override": "\t",
"data_dt_column": "date",
"data_dt_format": "%Y%m%d",
"data_dt_timezone": "UTC",
"data_key_column": "ticker"
}
}
```
## Composite key example
When the symbol is constructed from multiple columns:
```json theme={null}
{
"data_key_column": [
{ "type": "column", "value": "exchange" },
{ "type": "literal", "value": "_" },
{ "type": "column", "value": "ticker" }
]
}
```
This produces keys like `NYSE_AAPL`, `NASDAQ_MSFT`, etc.
## Multiple datetime columns
When the date and time are in separate columns:
```json theme={null}
{
"data_dt_column": ["trade_date", "trade_time"],
"data_dt_format": ["%Y-%m-%d", "%H:%M:%S.%f"]
}
```
CloudQuant Data Liberator concatenates the columns with a space before parsing, so the effective format becomes `"%Y-%m-%d %H:%M:%S.%f"`.
# SQL Server (MSSQL)
Source: https://knowledge.cloudquant.com/datasource-config/mssql
Configure Microsoft SQL Server datasources with ODBC support
# SQL Server (MSSQL)
SQL Server datasources use the ODBC driver and Microsoft ODBC Driver 18 for SQL Server.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on SQL Server connections.
## Connection configuration
### URI format
```
mssql://user:password@host:1433/database
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | ------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"mssql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | SQL Server connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (SQL Server default `1433`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Optional fields
| Field | Type | Default | Description |
| ------------ | ------ | ------- | ------------------------------------------------------- |
| `encryption` | string | | TLS encryption mode (e.g., `"yes"`, `"no"`, `"strict"`) |
### Environment variables
| Variable | Description |
| ------------------- | ----------------------------------------------------------------------------------------------- |
| `ODBC_MSSQL_DRIVER` | Override the ODBC driver name (default: `"ODBC Driver 18 for SQL Server"`) |
| `GODEBUG` | Set to `x509negativeserial=1` for servers using older certificates with negative serial numbers |
The `ODBC_MSSQL_DRIVER` environment variable allows you to use a different version of the Microsoft ODBC driver (e.g., Driver 17) without modifying the connection configuration.
### Example connection
```json theme={null}
{
"name": "mssql-market-data",
"connection_type": "mssql",
"connection_sub_type": "database",
"uri": "mssql://libuser:s3cureP@ss@db.example.com:1433/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss",
"encryption": "yes"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "mssql-trades",
"connection_type": "mssql",
"connection_sub_type": "database",
"uri": "mssql://libuser:s3cureP@ss@sqlserver.example.com:1433/trades_db",
"server": "sqlserver.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss",
"encryption": "yes"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "mssql-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "mssql-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native SQL Server `datetime2` or `datetimeoffset` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## Encryption configuration
SQL Server connections support several TLS encryption modes via the `encryption` field:
| Value | Description |
| ---------- | --------------------------------------------- |
| `"yes"` | Require encrypted connection (recommended) |
| `"no"` | Do not encrypt the connection |
| `"strict"` | Strict TLS 1.2+ encryption (SQL Server 2022+) |
When using `encryption: "yes"` with self-signed certificates, you may need to add `TrustServerCertificate=yes` to the connection. For production, always use properly signed certificates.
## Legacy certificate support
For SQL Server instances using older certificates with negative serial numbers:
```bash theme={null}
export GODEBUG=x509negativeserial=1
```
This is common with older SQL Server installations that have not rotated their TLS certificates.
## Driver and dependencies
| Component | Details |
| ---------------- | ------------------------------------------------ |
| Driver | `ODBC driver` with ODBC Driver 18 for SQL Server |
| Protocol | ODBC / TDS |
| Arrow conversion | High-performance ODBC driver |
| ODBC required | Yes (Microsoft ODBC Driver 18) |
| Default port | 1433 |
The Microsoft ODBC Driver 18 must be installed on the CloudQuant Data Liberator host. To use a different driver version, set the `ODBC_MSSQL_DRIVER` environment variable.
## Troubleshooting
### ODBC driver not found
Verify the Microsoft ODBC Driver 18 is installed and registered:
```bash theme={null}
# List installed ODBC drivers
odbcinst -q -d
# Expected output should include:
# [ODBC Driver 18 for SQL Server]
```
Install the driver if missing:
```bash Debian/Ubuntu theme={null}
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/22.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update && ACCEPT_EULA=Y apt-get install -y msodbcsql18
```
```bash RHEL/CentOS theme={null}
curl https://packages.microsoft.com/config/rhel/8/prod.repo > /etc/yum.repos.d/mssql-release.repo
ACCEPT_EULA=Y yum install -y msodbcsql18
```
### Using a different driver version
Override the default driver with an environment variable:
```bash theme={null}
export ODBC_MSSQL_DRIVER="ODBC Driver 17 for SQL Server"
```
### Certificate errors (x509)
If you see `x509: certificate signed by unknown authority` or negative serial number errors:
```bash theme={null}
export GODEBUG=x509negativeserial=1
```
### Connection timeout
SQL Server may be configured to listen on a non-default port or require a named instance. Verify connectivity:
```bash theme={null}
# Test port connectivity
nc -zv sqlserver.example.com 1433
# For named instances, use the SQL Browser port (1434/UDP) or specify the port directly
```
### Windows authentication
CloudQuant Data Liberator uses SQL Server authentication (username/password). Windows/Kerberos authentication is not supported through this connection type.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed.
* For partitioned tables, use `dt_regex` to scope queries to relevant partitions.
* Consider using `datetime2` over `datetime` for better precision and performance.
# MySQL
Source: https://knowledge.cloudquant.com/datasource-config/mysql
Configure MySQL datasources with ODBC support
# MySQL
MySQL datasources use the ODBC driver and the MariaDB ODBC connector. The connection string is built internally from the provided fields.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on MySQL connections.
## Connection configuration
### URI format
```
mysql://user:password@host:3306/database
```
The URI is used as a reference identifier. CloudQuant Data Liberator builds the actual ODBC connection string internally from the individual connection fields (`server`, `database`, `username`, `password`).
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | -------------------------------------------------------------------- |
| `connection_type` | string | Must be `"mysql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | MySQL connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (MySQL default `3306`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Optional fields
| Field | Type | Default | Description |
| -------------- | ------ | --------- | ------------------------------------------------------------ |
| `ssl_disabled` | string | `"False"` | Set to `"True"` to disable SSL. Values: `"True"` / `"False"` |
### Example connection
```json theme={null}
{
"name": "mysql-market-data",
"connection_type": "mysql",
"connection_sub_type": "database",
"uri": "mysql://libuser:s3cureP@ss@db.example.com:3306/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss",
"ssl_disabled": "False"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "mysql-trades",
"connection_type": "mysql",
"connection_sub_type": "database",
"uri": "mysql://libuser:s3cureP@ss@db.example.com:3306/trades_db",
"server": "db.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss",
"ssl_disabled": "False"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "mysql-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "mysql-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native MySQL `DATETIME` or `TIMESTAMP` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## SSL configuration
By default, SSL is enabled. To disable SSL (e.g., for local development):
```json theme={null}
{
"ssl_disabled": "True"
}
```
Disabling SSL in production is not recommended. Always use SSL when connecting over untrusted networks. The MariaDB ODBC driver supports TLS 1.2+ by default.
## Driver and dependencies
| Component | Details |
| ---------------- | ------------------------------------ |
| Driver | `ODBC driver` with MariaDB ODBC 3.2+ |
| Protocol | ODBC |
| Arrow conversion | High-performance ODBC driver |
| ODBC required | Yes (MariaDB ODBC connector) |
| Default port | 3306 |
The MariaDB ODBC 3.2+ driver must be installed on the CloudQuant Data Liberator host. This driver is compatible with both MySQL and MariaDB servers. Verify installation with `odbcinst -q -d`.
## Troubleshooting
### ODBC driver not found
If you see errors about missing ODBC drivers, verify the MariaDB ODBC driver is installed and registered:
```bash theme={null}
# List installed ODBC drivers
odbcinst -q -d
# Expected output should include:
# [MariaDB ODBC 3.2 Driver]
```
Install the driver if missing:
```bash Debian/Ubuntu theme={null}
apt-get install libmariadb3 odbc-mariadb
```
```bash RHEL/CentOS theme={null}
yum install MariaDB-connector-odbc
```
### Connection refused
Verify the MySQL server is accepting connections on the specified host and port. Check that the `bind-address` in the MySQL configuration allows remote connections.
```bash theme={null}
mysql -h db.example.com -P 3306 -u libuser -p marketdata
```
### Authentication failed
Ensure the user has been granted access from the CloudQuant Data Liberator host's IP address:
```sql theme={null}
GRANT SELECT ON trades_db.* TO 'libuser'@'liberator-host' IDENTIFIED BY 's3cureP@ss';
FLUSH PRIVILEGES;
```
### SSL handshake errors
If SSL connections fail, check that the MySQL server's SSL certificate is valid and trusted. Set `ssl_disabled` to `"True"` temporarily to confirm the issue is SSL-related.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in MySQL.
* For partitioned tables, ensure `dt_regex` is correctly scoping queries to relevant partitions.
# Oracle
Source: https://knowledge.cloudquant.com/datasource-config/oracle
Configure Oracle Database datasources with Oracle database driver thin mode
# Oracle
Oracle datasources use the `Oracle database driver` in thin mode. No Oracle Instant Client installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on Oracle connections.
## Connection configuration
### URI format
```
oracle://user:password@host:1521/service_name
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | --------------------------------------------------------------------- |
| `connection_type` | string | Must be `"oracle"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | Oracle connection string |
| `server` | string | Database host address |
| `database` | string | Oracle service name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (Oracle default `1521`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`). Missing tables return a clear error instead of an empty sample.
### Optional fields
| Field | Type | Default | Description |
| ----------- | ------ | ------- | --------------------------------------------- |
| `auth_mode` | string | | Oracle authentication mode (e.g., `"SYSDBA"`) |
| `schema` | string | | Oracle schema name to query against |
### Environment variables
| Variable | Description |
| ----------------------- | ----------------------------------------------- |
| `ORACLE_DEBUG_BOUNDARY` | Enable debug timing for Oracle query boundaries |
### Example connection
```json theme={null}
{
"name": "oracle-market-data",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://libuser:s3cureP@ss@db.example.com:1521/ORCL",
"server": "db.example.com",
"database": "ORCL",
"username": "libuser",
"password": "s3cureP@ss"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database/service name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "oracle-trades",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://libuser:s3cureP@ss@oradb.example.com:1521/TRADEDB",
"server": "oradb.example.com",
"database": "TRADEDB",
"username": "libuser",
"password": "s3cureP@ss",
"schema": "MARKET_DATA"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "oracle-trades",
"data_args": {
"table_options": [
{
"db": "TRADEDB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "oracle-trades",
"batch_size": 10000,
"data_dt_column": "TRADE_TIME",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "SYMBOL"
},
"schema": [
{ "name": "SYMBOL", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "TRADE_TIME", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "PRICE", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "VOLUME", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "EXCHANGE", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Oracle column names are uppercase by default. Use uppercase names in your schema definition and `data_args` fields to match Oracle's metadata.
## Authentication modes
The `auth_mode` field supports Oracle authentication modes for privileged connections:
```json theme={null}
{
"name": "oracle-admin",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://sys:adminP@ss@db.example.com:1521/ORCL",
"server": "db.example.com",
"database": "ORCL",
"username": "sys",
"password": "adminP@ss",
"auth_mode": "SYSDBA"
}
```
Using `SYSDBA` authentication in production datasource connections is not recommended. Create a dedicated read-only user with `SELECT` privileges on the required tables.
## Schema-qualified tables
When the `schema` field is set on the connection, CloudQuant Data Liberator queries tables within that schema. This is useful when the database user's default schema differs from where the data resides:
```json theme={null}
{
"schema": "MARKET_DATA",
"table_options": [
{
"db": "TRADEDB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
]
}
```
This queries `MARKET_DATA.DAILY_TRADES` rather than relying on the user's default schema.
## Microsecond timestamp precision
Oracle `TIMESTAMP(6)` columns support microsecond precision. A custom-patched `oracledb 3.4.0` is available for CloudQuant Data Liberator that preserves full microsecond precision during data extraction.
The standard `Oracle database driver 3.3+` driver supports microsecond timestamps. The custom-patched 3.4.0 build provides additional optimizations for high-frequency timestamp extraction. Contact your CloudQuant Data Liberator administrator for the patched driver.
## Driver and dependencies
| Component | Details |
| ---------------------- | ----------------------------------------- |
| Driver | `Oracle database driver` 3.3+ (thin mode) |
| Protocol | Oracle Net (TNS) |
| Arrow conversion | Via Oracle driver native conversion |
| Oracle Client required | No (thin mode) |
| Default port | 1521 |
Thin mode connects directly to Oracle without requiring Oracle Instant Client. This simplifies deployment and eliminates native library dependencies.
## Troubleshooting
### Connection refused
Verify the Oracle listener is running and accepting connections on the specified host and port:
```bash theme={null}
# Test port connectivity
nc -zv oradb.example.com 1521
# Test with sqlplus (if available)
sqlplus libuser/s3cureP@ss@oradb.example.com:1521/TRADEDB
```
### ORA-12514: TNS listener does not currently know of service
The service name in the URI does not match any registered service on the Oracle listener. Verify the service name:
```sql theme={null}
-- On the Oracle server
SELECT name FROM v$services;
```
### ORA-01017: invalid username/password
Ensure the username and password are correct. Oracle passwords are case-sensitive (depending on the `SEC_CASE_SENSITIVE_LOGON` parameter).
### Table not found (ORA-00942)
Check that the user has `SELECT` privileges on the target table and that the schema is correctly specified:
```sql theme={null}
GRANT SELECT ON MARKET_DATA.DAILY_TRADES TO libuser;
```
### Debug timing
To enable debug timing for Oracle query boundaries, set the environment variable:
```bash theme={null}
export ORACLE_DEBUG_BOUNDARY=1
```
This logs timing information for each query phase, useful for diagnosing performance bottlenecks.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in Oracle.
* For partitioned tables, verify that Oracle partition pruning is being used by checking the execution plan.
* Use `TIMESTAMP(6)` columns for microsecond precision rather than `DATE` (which only supports second precision).
# Datasource Configuration Overview
Source: https://knowledge.cloudquant.com/datasource-config/overview
Guide to configuring datasource connections in CloudQuant Data Liberator
# Datasource configuration
CloudQuant Data Liberator supports a wide range of datasource types for ingesting time series data. Each datasource requires a **connection** (how to reach the data) and a **dataset** (what data to extract and how to interpret it).
See [Supported Data Formats](/datasource-config/supported-formats) for the canonical list of file extensions and data source categories Liberator can ingest, including formats added after 2.0.
## Supported datasource types
### File-based sources
| Type | Description |
| --------------------------------------------------------------- | ------------------------------------------------------------------- |
| [Local File (CSV/TSV)](/datasource-config/local-file) | Flat files on local/mounted storage |
| [S3](/datasource-config/s3) | Amazon S3 or S3-compatible object storage |
| [Azure Blob Storage](/datasource-config/azure-blob) | Microsoft Azure Blob containers |
| [SharePoint / OneDrive](/datasource-config/sharepoint-onedrive) | Microsoft 365 file storage via share link (2.2+); wildcards in 2.3+ |
| [SFTP](/datasource-config/sftp) | SSH File Transfer Protocol servers |
| [FTPS](/datasource-config/ftps) | FTP over TLS/SSL |
| [CIFS/SMB](/datasource-config/cifs) | Windows/Samba network file shares |
### Database sources
| Type | Description |
| ------------------------------------------- | -------------------------------------- |
| [PostgreSQL](/datasource-config/postgresql) | High-performance native driver |
| [MySQL](/datasource-config/mysql) | Via ODBC driver (MySQL-compatible) |
| [SQL Server](/datasource-config/mssql) | Via ODBC driver (ODBC Driver 18) |
| [Oracle](/datasource-config/oracle) | Via Oracle database driver (thin mode) |
| [Snowflake](/datasource-config/snowflake) | High-performance native driver |
### API sources
| Type | Description |
| ------------------------------------- | ---------------------------------- |
| [HubSpot](/datasource-config/hubspot) | HubSpot CRM objects via API (2.3+) |
### File and data formats
Beyond the delimited text examples in each connection guide, Liberator also ingests Parquet, Arrow IPC, Excel, XML, HDF5, ZIP archives, database tables, and API payloads. PSV (2.1+) and PCAP/PCAPng FIX capture (2.2+) are documented in [Supported Data Formats](/datasource-config/supported-formats).
## Architecture: connection + dataset
Every datasource in CloudQuant Data Liberator is composed of two parts:
### Connection
Defines **how to reach** the data — credentials, endpoints, paths, and transport protocol.
```
Connection → "Where is the data and how do I authenticate?"
```
### Dataset
Defines **what to extract** — which table/files, timestamp columns, key columns, schema, and data frequency.
```
Dataset → "What data do I want and how do I interpret it?"
```
In Liberator **2.3** and later, you can create both through the **AI Assisted** Create Dataset flow. Describe the source in plain language (or paste connection details), then review the auto-filled wizard steps before creating. See [AI-driven dataset onboarding](/administration/ai-dataset-onboarding).
## Common configuration concepts
### Timestamp configuration
All datasources require timestamp configuration to map source data into CloudQuant Data Liberator's microsecond timestamp (`muts`) format:
| Field | Description |
| ------------------ | ----------------------------------------------------------------- |
| `data_dt_column` | Column(s) containing the datetime |
| `data_dt_format` | Format string or parsing specification |
| `data_dt_timezone` | Timezone of the source data (e.g., `"UTC"`, `"America/New_York"`) |
| `data_dt_nudge` | Microsecond offset applied to timestamps |
#### Supported datetime formats
| Format | Description |
| --------------------- | ---------------------------------------------- |
| `"%Y-%m-%d %H:%M:%S"` | Standard strptime format |
| `"datetime"` | Native database datetime column |
| `"date"` | Native date column (date32/date64) |
| `"muts"` | Unix epoch microseconds |
| `"uts"` | Unix epoch seconds |
| `"nuts"` | Unix epoch nanoseconds |
| `true` | Auto-detect native datetime (database sources) |
### Key column configuration
The `data_key_column` field defines the symbol/key used for filtering queries:
```python theme={null}
# Simple string
"data_key_column": "symbol"
# Composite key with literals and columns
"data_key_column": [
{"type": "column", "value": "exchange"},
{"type": "literal", "value": "_"},
{"type": "column", "value": "ticker"}
]
```
### Schema definition
Each column in a dataset schema requires:
```json theme={null}
{
"name": "column_name",
"type": "int64",
"group": "value",
"description": "Human-readable description",
"display_name": "Display Name"
}
```
**Column types:** `string`, `int64`, `uint64`, `double`, `float`, `bool`, `date32`, `date64`, `time64`
**Column groups:**
* `key` — Symbol/key columns
* `time` — Timestamp columns
* `value` — Data columns
* `meta` — System columns (`_seq`, `muts`, etc.)
### Auto-generated columns
CloudQuant Data Liberator automatically generates these columns if not present in source data:
| Column | Type | Description |
| ----------- | ------ | -------------------------------------------- |
| `_seq` | uint64 | Sequential row number within partition |
| `muts` | int64 | Microseconds since Unix epoch |
| `timestamp` | string | Human-readable timestamp (America/New\_York) |
| `symbol` | string | Key column (copied from `data_key_column`) |
### File name date extraction
For file-based sources, dates can be extracted from filenames:
| Field | Description | Example |
| ------------------------- | --------------------------------------- | ------------------------------- |
| `fname_dt_regex` | Regex to match date portion of filename | `data_(\d{4}-\d{2}-\d{2})\.csv` |
| `fname_dt_format` | strptime format for the matched portion | `%Y-%m-%d` |
| `fname_dt_timezone` | Timezone of the filename date | `UTC` |
| `fname_dt_nudge` | Microsecond offset | `0` |
| `fname_dt_approx_seconds` | Approximate seconds per file | `86400` |
# PostgreSQL
Source: https://knowledge.cloudquant.com/datasource-config/postgresql
Configure PostgreSQL datasources with high-performance native driver support
# PostgreSQL
PostgreSQL datasources use a high-performance native driver for optimized data transfer. No ODBC installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on PostgreSQL connections.
## Connection configuration
### URI format
```
postgresql://user:password@host:5432/database
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | ------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"postgresql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | PostgreSQL connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (PostgreSQL default `5432`) |
The native driver communicates with the database using the PostgreSQL binary protocol and produces data batches directly, avoiding row-by-row conversion overhead. In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Example connection
```json theme={null}
{
"name": "postgres-market-data",
"connection_type": "postgresql",
"connection_sub_type": "database",
"uri": "postgresql://libuser:s3cureP@ss@db.example.com:5432/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "postgres-trades",
"connection_type": "postgresql",
"connection_sub_type": "database",
"uri": "postgresql://libuser:s3cureP@ss@db.example.com:5432/trades_db",
"server": "db.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "postgres-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "postgres-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native PostgreSQL `timestamp` or `timestamptz` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## Partitioned table example
For tables partitioned by date (e.g., `trades_2024_01`, `trades_2024_02`):
```json theme={null}
{
"table_options": [
{
"db": "trades_db",
"table_like": "trades_%",
"token": "trades",
"zone": "America/New_York",
"dt_regex": "trades_(\\d{4})_(\\d{2})",
"dt_format": "%Y_%m"
}
]
}
```
## Driver and dependencies
| Component | Details |
| --------------- | -------------------------- |
| Driver | PostgreSQL native driver |
| Protocol | PostgreSQL binary protocol |
| Data conversion | Native (high-performance) |
| ODBC required | No |
| Default port | 5432 |
The native driver is bundled with CloudQuant Data Liberator. No additional driver installation is needed on the host system.
## Troubleshooting
### Connection refused
Verify the PostgreSQL server is accepting connections on the specified host and port. Check `pg_hba.conf` for client authentication rules.
```bash theme={null}
psql -h db.example.com -p 5432 -U libuser -d marketdata
```
### Authentication failed
Ensure the `username` and `password` in the connection match a valid PostgreSQL role. Check that the role has `CONNECT` privilege on the database and `SELECT` privilege on the target tables.
### SSL/TLS errors
If the server requires SSL, append `?sslmode=require` to the URI:
```
postgresql://libuser:s3cureP@ss@db.example.com:5432/marketdata?sslmode=require
```
Using `sslmode=disable` in production is not recommended. Always use `sslmode=require` or `sslmode=verify-full` when connecting over untrusted networks.
### Empty results from partitioned tables
Verify that `dt_regex` correctly matches the table name suffix and that `dt_format` corresponds to the captured groups. Test the regex against your actual table names.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in PostgreSQL.
* Use `table_like` to target specific tables rather than scanning all partitions.
# Amazon S3
Source: https://knowledge.cloudquant.com/datasource-config/s3
Configure S3 or S3-compatible object storage datasources
# Amazon S3
S3 datasources allow CloudQuant Data Liberator to read CSV, TSV, and Parquet files directly from Amazon S3 buckets or S3-compatible object storage services (MinIO, Wasabi, Backblaze B2, etc.).
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest on S3, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------------- | ------ | --------------------------------------------------- |
| `connection_type` | string | Must be `"s3"` |
| `aws_access_key_id` | string | AWS access key ID |
| `aws_secret_access_key` | string | AWS secret access key |
| `bucket` | string | S3 bucket name |
| `endpoint` | string | S3 endpoint URL, e.g., `"https://s3.amazonaws.com"` |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ------ | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `prefix` | string | `""` | Key prefix (virtual directory) within the bucket |
| `request_style` | string | auto (2.4+) | S3 addressing: `"path"` or `"virtual"`. In Liberator **2.4** and later, Test Connection probes the endpoint and persists the correct style. Override only if the probe is wrong. |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
| `object_key` | string | | Object key pattern for file selection |
In Liberator **2.4** and later, Test Connection auto-detects path-style vs virtual-hosted addressing for AWS and S3-compatible endpoints (MinIO, Ceph, Wasabi, Oracle Cloud, and custom hosts). You can still set `request_style` explicitly. For older releases, set `request_style` to `"path"` for most S3-compatible services and keep `"virtual"` for AWS S3.
### Example connection
```json theme={null}
{
"name": "s3-market-data",
"connection_type": "s3",
"aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
"aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"bucket": "my-market-data",
"endpoint": "https://s3.amazonaws.com",
"prefix": "daily/equities/",
"request_style": "virtual"
}
```
Never commit AWS credentials to version control. Use environment variables or a secrets manager to inject credentials at deployment time.
## CSV/TSV dataset
The `data_args` are identical to [Local File](/datasource-config/local-file) sources. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
### Complete CSV example
```json theme={null}
{
"name": "s3-equity-trades",
"connection": "s3-market-data",
"data_args": {
"file_pattern": "trades_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "timestamp",
"data_dt_format": "%Y-%m-%d %H:%M:%S.%f",
"data_dt_timezone": "UTC",
"data_key_column": "symbol",
"fname_dt_regex": "trades_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "UTC",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "timestamp", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "size", "type": "int64", "group": "value", "description": "Trade size" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
## Parquet dataset
This provides passthrough access to Parquet data without intermediate caching, leveraging Arrow's native Parquet reader.
Parquet files offer zero-copy reads and columnar pushdown. This is significantly more efficient than converting Parquet to CSV.
### Complete Parquet example
```json theme={null}
{
"name": "s3-equity-bars-parquet",
"connection": "s3-market-data",
"data_args": {
"file_pattern": "bars_*.parquet",
"data_dt_column": "bar_time",
"data_dt_format": "datetime",
"data_dt_timezone": "America/New_York",
"data_key_column": "symbol",
"fname_dt_regex": "bars_(\\d{4}-\\d{2}-\\d{2})\\.parquet",
"fname_dt_format": "%Y-%m-%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "bar_time", "type": "string", "group": "time", "description": "Bar timestamp" },
{ "name": "open", "type": "double", "group": "value", "description": "Open price" },
{ "name": "high", "type": "double", "group": "value", "description": "High price" },
{ "name": "low", "type": "double", "group": "value", "description": "Low price" },
{ "name": "close", "type": "double", "group": "value", "description": "Close price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Bar volume" }
]
}
```
## S3-compatible storage
### MinIO example
```json theme={null}
{
"name": "minio-connection",
"connection_type": "s3",
"aws_access_key_id": "minio-access-key",
"aws_secret_access_key": "minio-secret-key",
"bucket": "market-data",
"endpoint": "https://minio.internal.example.com:9000",
"request_style": "path"
}
```
Most S3-compatible services require `request_style` set to `"path"`. Only AWS S3 defaults to virtual-hosted style.
## IAM permissions
The IAM user or role associated with the access key needs at minimum:
* `s3:GetObject` on the bucket objects
* `s3:ListBucket` on the bucket
See the [S3 Bucket Setup](/integrations/s3-bucket-setup) guide for detailed IAM policy configuration.
# SFTP
Source: https://knowledge.cloudquant.com/datasource-config/sftp
Configure SFTP datasources
# SFTP
SFTP (SSH File Transfer Protocol) datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from remote servers over an encrypted SSH connection. CloudQuant Data Liberator mounts the remote directory via SSHFS/FUSE.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest over SFTP, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | ---------------------------------- |
| `connection_type` | string | Must be `"sftp"` |
| `host` | string | SFTP server hostname or IP address |
| `user` | string | Username for authentication |
You must provide either `password` or `key` for authentication. If both are specified, key-based authentication takes precedence.
### Optional fields
| Field | Type | Default | Description |
| ------------- | ------ | ------- | ----------------------------------------------------------------- |
| `port` | int | `22` | SSH port number |
| `password` | string | | Password for password-based authentication |
| `key` | string | | SSH private key content (PEM format) for key-based authentication |
| `prefix` | string | `""` | Remote directory path to use as root |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
### Example connection (password authentication)
```json theme={null}
{
"name": "sftp-vendor-data",
"connection_type": "sftp",
"host": "sftp.vendor.example.com",
"port": 22,
"user": "datauser",
"password": "s3cur3P@ssw0rd",
"prefix": "/data/daily-feeds/"
}
```
### Example connection (key authentication)
```json theme={null}
{
"name": "sftp-internal-data",
"connection_type": "sftp",
"host": "data-server.internal.net",
"port": 2222,
"user": "liberator-svc",
"key": "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNza...\n-----END OPENSSH PRIVATE KEY-----",
"prefix": "/exports/market-data/"
}
```
Avoid embedding private keys or passwords directly in configuration files. Use environment variables or a secrets manager to inject credentials at deployment time.
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "sftp-trades-feed",
"connection_type": "sftp",
"host": "sftp.dataprovider.com",
"port": 22,
"user": "cq-ingest",
"password": "vendorPassword123",
"prefix": "/feeds/trades/"
}
```
### Dataset
```json theme={null}
{
"name": "vendor-trades",
"connection": "sftp-trades-feed",
"data_args": {
"file_pattern": "trades_*.csv.gz",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": ["date", "time"],
"data_dt_format": ["%Y%m%d", "%H:%M:%S.%f"],
"data_dt_timezone": "America/New_York",
"data_key_column": "symbol",
"fname_dt_regex": "trades_(\\d{8})\\.csv\\.gz",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "date", "type": "string", "group": "time", "description": "Trade date" },
{ "name": "time", "type": "string", "group": "time", "description": "Trade time" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "size", "type": "int64", "group": "value", "description": "Trade size" },
{ "name": "condition", "type": "string", "group": "value", "description": "Sale condition code" }
]
}
```
CloudQuant Data Liberator supports reading gzip-compressed files (`.csv.gz`) transparently. Use compressed files on SFTP connections to reduce transfer time over slow or high-latency links.
## Network requirements
Ensure the following network connectivity from the CloudQuant Data Liberator host:
| Requirement | Detail |
| ------------------ | ------------------------------------------------------------------ |
| **Outbound port** | TCP port 22 (or custom port) to the SFTP server |
| **DNS resolution** | The hostname must resolve from the CloudQuant Data Liberator host |
| **Firewall rules** | Whitelist the CloudQuant Data Liberator host IP on the SFTP server |
| **SSH host key** | The server's host key must be trusted (added to known\_hosts) |
# SharePoint and OneDrive
Source: https://knowledge.cloudquant.com/datasource-config/sharepoint-onedrive
Connect SharePoint or OneDrive data sources using a Microsoft share link
# SharePoint and OneDrive
SharePoint and OneDrive connections let Liberator read delimited files, Parquet, Excel, and other [supported formats](/datasource-config/supported-formats) from Microsoft 365 storage.
In Liberator 2.2 and later, paste a standard **share link** copied from the browser or desktop to connect. In Liberator **2.3** and later, wildcard folder and file patterns (for example, `reports/2026-*/trades_*.csv`) are supported for both SharePoint and OneDrive sources.
## Connect with a share link
In the browser or desktop app, open the file or folder, then **Share → Copy link**. The link looks like `https://your-org.sharepoint.com/:f:/s/...`.
Go to **Connections → Add Connection** and select **SharePoint** or **OneDrive**.
Paste the link into the **Share link** field. Liberator clears any previously entered folder path when a share link is detected.
Complete the **Sign in with Microsoft** dialog. If it does not appear automatically, click **Sign in with Microsoft**.
Click **Test Connection**. On success, the resolved internal path replaces the share link in the field. Confirm the account has at least read access to the linked location.
Continue through dataset configuration (file pattern, columns, date/time) as for other file-based sources.
## Fixed date and time (non-time-series files)
For static reference files without a date column:
1. On the **Date / Time** step, select **Fixed date and time**.
2. Enter the timestamp Liberator should assign to records.
If the source has a date column, select it from the **Date column** dropdown instead.
## Date format picker (2.2+)
When choosing a date format during onboarding, Liberator:
* Shows a **live example from your sample data** (for example, `25 Jun 2026` instead of generic `DD MMM YYYY`).
* **Ranks formats by likelihood**, with the best match at the top.
Select the example that matches your source dates.
## Related
File extensions available on SharePoint and OneDrive connections
Shared `data_args` fields for file-based datasets
# Snowflake
Source: https://knowledge.cloudquant.com/datasource-config/snowflake
Configure Snowflake datasources with native high-performance driver
# Snowflake
Snowflake datasources use the Snowflake native driver for high-performance native data transfer. No ODBC installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on Snowflake connections.
## Connection configuration
### URI format
```
snowflake://user:password@account/database/schema?warehouse=compute_wh&role=accountadmin
```
### Required fields
| Field | Type | Description |
| --------------------- | ------ | --------------------------------------------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"snowflake"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | Snowflake connection string |
| `server` | string | Snowflake account URL (e.g., `"xy12345.us-east-1.snowflakecomputing.com"`) |
| `database` | string | Snowflake database name |
| `username` | string | Snowflake user |
| `password` | string | Snowflake password. In Liberator **2.4+**, special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`) are supported |
| `account` | string | Snowflake account identifier (e.g., `"xy12345.us-east-1"`) |
| `warehouse` | string | Compute warehouse name |
| `role` | string | Snowflake role |
### Optional fields
| Field | Type | Default | Description |
| -------- | ------ | ------- | --------------------- |
| `schema` | string | | Snowflake schema name |
### Environment variables
| Variable | Description |
| --------------------- | ------------------------------------------------------ |
| `SNOWFLAKE_DEBUGGING` | Enable verbose debug logging for Snowflake connections |
### Example connection
```json theme={null}
{
"name": "snowflake-market-data",
"connection_type": "snowflake",
"connection_sub_type": "database",
"uri": "snowflake://libuser:s3cureP@ss@xy12345.us-east-1/MARKETDATA/PUBLIC?warehouse=COMPUTE_WH&role=DATA_READER",
"server": "xy12345.us-east-1.snowflakecomputing.com",
"database": "MARKETDATA",
"username": "libuser",
"password": "s3cureP@ss",
"account": "xy12345.us-east-1",
"warehouse": "COMPUTE_WH",
"role": "DATA_READER",
"schema": "PUBLIC"
}
```
The Snowflake native driver uses Snowflake's native result set format. Data is returned as batches directly from the Snowflake service with no row-by-row conversion.
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "snowflake-trades",
"connection_type": "snowflake",
"connection_sub_type": "database",
"uri": "snowflake://libuser:s3cureP@ss@xy12345.us-east-1/TRADES_DB/MARKET?warehouse=COMPUTE_WH&role=accountadmin",
"server": "xy12345.us-east-1.snowflakecomputing.com",
"database": "TRADES_DB",
"username": "libuser",
"password": "s3cureP@ss",
"account": "xy12345.us-east-1",
"warehouse": "COMPUTE_WH",
"role": "accountadmin",
"schema": "MARKET"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "snowflake-trades",
"data_args": {
"table_options": [
{
"db": "TRADES_DB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "snowflake-trades",
"batch_size": 10000,
"data_dt_column": "TRADE_TIME",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "SYMBOL"
},
"schema": [
{ "name": "SYMBOL", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "TRADE_TIME", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "PRICE", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "VOLUME", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "EXCHANGE", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Snowflake identifiers are uppercase by default. Use uppercase names in your schema definition and `data_args` fields to match Snowflake's metadata, unless the objects were created with double-quoted lowercase names.
## Warehouse and role configuration
The `warehouse` and `role` fields control compute resources and access permissions:
### Warehouse
The warehouse determines the compute cluster used to execute queries. Choose a warehouse sized appropriately for your query workload:
```json theme={null}
{
"warehouse": "COMPUTE_WH"
}
```
Ensure the specified warehouse is set to auto-resume, or queries will fail when the warehouse is suspended. CloudQuant Data Liberator does not issue `ALTER WAREHOUSE ... RESUME` commands.
### Role
The role determines which database objects are accessible:
```json theme={null}
{
"role": "DATA_READER"
}
```
Create a dedicated read-only role for CloudQuant Data Liberator connections rather than using `accountadmin`. Grant `SELECT` on the required databases and schemas to this role.
## Debug logging
Enable verbose debug logging for Snowflake connections to diagnose connectivity or query issues:
```bash theme={null}
export SNOWFLAKE_DEBUGGING=1
```
This produces detailed logs including connection negotiation, query execution timing, and data batch transfer metrics.
## Driver and dependencies
| Component | Details |
| --------------- | --------------------------------------- |
| Driver | `Snowflake native driver` |
| Protocol | Snowflake REST API / streaming protocol |
| Data conversion | Native (high-performance) |
| ODBC required | No |
| Default port | 443 (HTTPS) |
The Snowflake native driver is bundled with CloudQuant Data Liberator. No additional driver installation is needed on the host system. All communication uses HTTPS on port 443.
## Troubleshooting
### Account identifier format
The `account` field must include the region if your account is not in the default AWS us-west-2 region:
```
# US East (Ohio)
xy12345.us-east-1
# EU (Frankfurt)
xy12345.eu-central-1
# Azure (East US 2)
xy12345.east-us-2.azure
```
### Authentication failed (390100)
Verify the username, password, and account identifier. Snowflake passwords are case-sensitive. If multi-factor authentication (MFA) is enabled for the user, you may need to use a service account without MFA.
### Warehouse is suspended
If queries fail with a warehouse suspension error, ensure the warehouse is configured with `AUTO_RESUME = TRUE`:
```sql theme={null}
ALTER WAREHOUSE COMPUTE_WH SET AUTO_RESUME = TRUE;
```
### Role does not have access
Verify the role has the required privileges:
```sql theme={null}
GRANT USAGE ON DATABASE TRADES_DB TO ROLE DATA_READER;
GRANT USAGE ON SCHEMA TRADES_DB.MARKET TO ROLE DATA_READER;
GRANT SELECT ON ALL TABLES IN SCHEMA TRADES_DB.MARKET TO ROLE DATA_READER;
```
### Network connectivity
Snowflake connections require HTTPS (port 443) access to `*.snowflakecomputing.com`. Ensure firewalls and proxy servers allow this traffic.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure Snowflake clustering keys align with your datetime and key columns.
* Use an appropriately sized warehouse (e.g., `MEDIUM` or `LARGE`) for large datasets.
* Check that the warehouse is not queued behind other workloads using Snowflake's query history.
# Supported Data Formats
Source: https://knowledge.cloudquant.com/datasource-config/supported-formats
File and data formats CloudQuant Data Liberator can ingest, by platform version
# Supported data formats
CloudQuant Data Liberator ingests time-series and tabular data through **connections** (where data lives) and **datasets** (how files or tables are interpreted). The tables below list every **file extension** and **data source category** the platform understands.
Formats are selected automatically from your `file_pattern`, file extension, or connection type when you onboard through the Liberator UI. You rarely need to set anything manually.
This page is the canonical format reference. Each [connection guide](/datasource-config/overview) links here for the formats available on that storage type.
## Supported in version 2.0
These formats were available in the Liberator **2.0** release and remain supported on current versions.
### Delimited text files
| Extension | Delimiter | Configuration |
| --------- | --------------- | ----------------------------------------------------- |
| `.csv` | Comma (default) | `file_pattern` such as `*.csv` |
| `.tsv` | Tab | `file_pattern` such as `*.tsv` |
| `.txt` | Auto-detected | Same pipeline as CSV; delimiter inferred when omitted |
Set `sep_override` in `data_args` to force comma, tab, pipe (`\|`), or semicolon when auto-detection is not sufficient.
### Columnar and binary files
| Extension | Description |
| -------------------- | ------------------------------------------------- |
| `.parquet` | Apache Parquet; columnar reads with type pushdown |
| `.arrow`, `.feather` | Apache Arrow IPC (Feather v2) |
On S3, Parquet datasets can use **passthrough** mode (direct read without intermediate caching). See the [S3 Parquet example](/datasource-config/s3#parquet-dataset).
### Structured office and scientific files
| Extension | Description | Extra configuration |
| --------------- | ------------------------- | ---------------------------------------------------- |
| `.xlsx`, `.xls` | Microsoft Excel workbooks | First sheet sampled at onboarding |
| `.xml` | XML documents | `xml_args` for element paths (set during onboarding) |
| `.h5`, `.hdf5` | HDF5 scientific arrays | `h5py_groups` for dataset path inside the file |
### Archives
| Extension | Description |
| --------- | --------------------------------------------------------------------------------------- |
| `.zip` | ZIP archive; Liberator inspects the inner file and applies the matching format delegate |
### Database sources
| Connection type | Engine | Access pattern |
| --------------- | ------------------------------ | ------------------------- |
| PostgreSQL | Native high-performance driver | Table or view per dataset |
| MySQL | ODBC (MariaDB-compatible) | Table or view per dataset |
| SQL Server | ODBC Driver 18 | Table or view per dataset |
| Oracle | Thin driver | Table or view per dataset |
| Snowflake | Native driver | Table or view per dataset |
### API-backed datasets
REST endpoints that return JSON tabular payloads can be onboarded as **API** connections. The platform normalizes responses into the same query surface as file- and database-backed datasets.
### Schema column types
Regardless of source format, dataset schemas use these column types:
`string`, `int64`, `uint64`, `double`, `float`, `bool`, `date32`, `date64`, `time64`
See [Datasource configuration overview](/datasource-config/overview#schema-definition) for column groups (`key`, `time`, `value`, `meta`).
## Formats added after version 2.0
The following ingest formats were added in subsequent releases. They are available on current CloudQuant-managed environments at those versions and later.
### Version 2.1 — PSV (pipe-separated values)
| Extension | Delimiter | Notes |
| --------- | ----------- | ----------------------------------------------- |
| `.psv` | Pipe (`\|`) | First-class extension alongside CSV and Parquet |
PSV files use the same onboarding flow as CSV: header row, per-column type inference, and configurable null sentinel. You can also ingest pipe-delimited `.csv` or `.txt` files by setting `sep_override` to `"\|"` without renaming the file.
### Version 2.2 — PCAP / PCAPng (FIX tick capture)
| Extension | Description |
| --------- | ------------------------------ |
| `.pcap` | Classic packet capture |
| `.pcapng` | Next-generation packet capture |
Liberator extracts **FIX-protocol messages** from TCP payloads in packet captures and exposes them through the standard query API. Typical columns include FIX tags such as `35` (MsgType), `49` (SenderCompID), `52` (SendingTime), and `55` (Symbol), plus `_pcap_ts_ns` for the capture timestamp.
Use PCAP datasets when you capture exchange feeds at the wire level and want the same query model as historical bar or trade datasets.
## Added in version 2.3
No new file or packet-capture ingest formats ship in 2.3. Connectivity additions in this release (HubSpot CRM objects, OneDrive/SharePoint wildcards) are covered in the [HubSpot](/datasource-config/hubspot) and [SharePoint and OneDrive](/datasource-config/sharepoint-onedrive) guides and in [What's New in Liberator 2.3](/whats-new/liberator-2.3).
## Added in version 2.4
No new file or packet-capture ingest formats ship in 2.4. Query, catalog, and connection changes in this release (symbol pattern filters, dataset health cards, S3 request-style auto-detect, required database port) are covered in [What's New in Liberator 2.4](/whats-new/liberator-2.4).
## Where each format applies
All **file-based** formats in the tables above can be stored on any file-backed connection type:
| Connection | Guide |
| ----------------------------- | ------------------------------------------- |
| Local / mounted directory | [Local File](/datasource-config/local-file) |
| Amazon S3 (and S3-compatible) | [S3](/datasource-config/s3) |
| Azure Blob Storage | [Azure Blob](/datasource-config/azure-blob) |
| SFTP | [SFTP](/datasource-config/sftp) |
| FTPS | [FTPS](/datasource-config/ftps) |
| CIFS / SMB | [CIFS](/datasource-config/cifs) |
**Database** and **API** formats map to their respective connection guides:
| Connection | Guide |
| ---------- | ------------------------------------------- |
| PostgreSQL | [PostgreSQL](/datasource-config/postgresql) |
| MySQL | [MySQL](/datasource-config/mysql) |
| SQL Server | [SQL Server](/datasource-config/mssql) |
| Oracle | [Oracle](/datasource-config/oracle) |
| Snowflake | [Snowflake](/datasource-config/snowflake) |
## Choosing a format
| Use case | Recommended format |
| ---------------------------------------------------- | ------------------------------------- |
| Human-readable exports from spreadsheets or ETL jobs | CSV or TSV |
| Vendor pipe-delimited daily drops | PSV (2.1+) or CSV with `sep_override` |
| Large historical archives, column pruning | Parquet |
| Low-latency interchange between Arrow-native tools | Arrow IPC / Feather |
| Excel exports from business users | XLSX |
| Scientific simulation output | HDF5 |
| Hierarchical vendor XML feeds | XML |
| SQL warehouse tables already in your estate | Matching database connection |
| Wire-level FIX tick replay | PCAP / PCAPng (2.2+) |
## Related
Connection + dataset architecture and shared `data_args` fields
Full `data_args` reference for file-based sources
# Azure Blob Storage
Source: https://knowledge.cloudquant.com/integrations/azure-blob-storage
Retrieving access keys for Azure Blob Storage connections with CloudQuant Data Liberator
# Azure Blob Storage connections
This guide explains how to retrieve and configure Azure Blob Storage access keys for use with the CloudQuant Data Liberator platform.
## Prerequisites
* An Azure account with a Storage Account
* Access to the Azure Portal
* Your CloudQuant account credentials
## Retrieving access keys
### Step 1: Navigate to your storage account
1. Sign in to the [Azure Portal](https://portal.azure.com)
2. Navigate to **Storage accounts**
3. Select the storage account you want to connect
### Step 2: Find access keys
1. In the storage account menu, select **Security + networking** > **Access keys**
2. You will see two keys: **key1** and **key2**
3. Select **Show** to reveal the key values
### Step 3: Copy required information
You will need:
* **Storage account name** — The name of your Azure Storage Account
* **Access key** — Either key1 or key2
* **Container name** — The blob container holding your data
Keep your access keys secure. Never commit them to source control or share them in plain text.
## Configuring the connection
Provide the following to CloudQuant for CloudQuant Data Liberator connectivity:
```
Storage Account: your_storage_account_name
Access Key: your_access_key
Container: your_container_name
```
## Connection string format
If configuring programmatically, the connection string format is:
```
DefaultEndpointsProtocol=https;AccountName=your_account;AccountKey=your_key;EndpointSuffix=core.windows.net
```
## Key rotation
Azure provides two keys to enable rotation without downtime:
1. Update your applications to use **key2**
2. Regenerate **key1**
3. Update applications to use the new **key1**
4. Regenerate **key2**
Rotate your access keys regularly as a security best practice.
## Troubleshooting
| Issue | Solution |
| --------------------- | ----------------------------------------------------------------- |
| Authentication failed | Verify the access key hasn't been rotated |
| Container not found | Check the container name and ensure it exists |
| Network error | Verify firewall rules allow access from CloudQuant Data Liberator |
# ChatGPT
Source: https://knowledge.cloudquant.com/integrations/chatgpt
Connect ChatGPT to CloudQuant Data Liberator with the MCP connector and OAuth authentication
# Connecting Liberator to ChatGPT
Use ChatGPT's MCP connector with **OAuth** to query CloudQuant Data Liberator from a ChatGPT workspace. Each user authenticates with their own Liberator credentials, so data access follows individual entitlements rather than a shared API key.
## Prerequisites
* Access through [app.cloudquant.ai](https://app.cloudquant.ai) — ChatGPT MCP setup for Liberator is done from that CloudQuant environment.
* A **ChatGPT Pro or Plus** account — the MCP connector is only available on paid tiers.
* **Developer tool access** enabled in ChatGPT (required before you can create custom Apps). Enable it under **Settings**.
* A CloudQuant Data Liberator account with valid username and password.
* The **Liberator MCP Endpoint URL** from the Liberator UI (for example, `https://api.cloudquant.ai/mcp`).
OAuth is the recommended authentication method. Each user who connects through the ChatGPT app authenticates with their own Liberator credentials, so entitlements stay per-user.
## Setup steps
In ChatGPT, click your workspace or profile name in the lower-left corner, then select **Settings**.
In Settings, open the **Advanced** section, click **Apps**, then click **Create App**.
In the New App dialog, set:
| Field | Value |
| --------------- | ----------------------------------------------------------------------------------------------- |
| **Name** | `CloudQuant Liberator` |
| **Description** | `MCP Connector for CloudQuant Data Liberator` |
| **Connection** | Select **Server URL** (not Tunnel). Enter the Liberator MCP Endpoint URL from the Liberator UI. |
Example MCP Endpoint URL:
```text theme={null}
https://api.cloudquant.ai/mcp
```
Copy the exact URL from your Liberator UI if your environment differs.
Under **Authentication**, select **OAuth**. This is the recommended method for Liberator and keeps access tied to each user's credentials.
ChatGPT shows a warning that custom MCP servers introduce risk. Check **I understand and want to continue**, then click **Create**. The Create button stays inactive until you acknowledge the warning.
ChatGPT starts the OAuth flow and shows a **Grant Access to ChatGPT** screen. Approve access for:
* User profile
* `openid`
* User roles
* Email address
* `groups`
Click **Yes** to grant access so ChatGPT can authenticate to the Liberator MCP server on your behalf.
Once access is granted, the CloudQuant Liberator MCP connector is ready to use in your ChatGPT workspace.
## Using the connector
After the app is created, Liberator MCP tools are available in ChatGPT conversations:
| Tool | Purpose |
| -------------------- | ------------------------------------------------- |
| `list_datasets` | List available Liberator datasets |
| `get_dataset_config` | Fetch configuration and schema for a dataset |
| `query_dataset` | Query a Liberator dataset by name with parameters |
Because OAuth is used, each user who interacts with the connector is prompted to authenticate with their own CloudQuant credentials. Individual data access and entitlements apply — no shared API keys are required.
## Related
Add Liberator as an MCP tool in a Copilot Studio agent
Install the Liberator .mcpb extension in Claude Desktop
Liberator 2.3 release notes, including AI-driven reporting
Connect Liberator metrics to Grafana with a Bearer token
# Claude Desktop
Source: https://knowledge.cloudquant.com/integrations/claude
Connect Claude Desktop to CloudQuant Data Liberator with the MCP extension bundle
# Connecting Liberator to Claude Desktop
Install the Liberator extension bundle (`.mcpb`) in Claude Desktop to query Liberator datasets directly from chat through the Model Context Protocol (MCP).
## Prerequisites
* [Claude Desktop](https://claude.ai/download) installed on your machine
* A CloudQuant Data Liberator account
* The Liberator **MCP Endpoint URL** from the Liberator UI (for example, `https://api.cloudquant.ai/mcp`)
* The Liberator **`.mcpb` extension bundle**, downloadable from the Liberator UI **Profile → Documentation → A.I.** section (or from `https:///permissions/download-liberator-client/?lang=mcpb` while signed in)
## Setup steps
In the Liberator UI, open **Profile → Documentation → A.I. → Claude Desktop** and download the `.mcpb` file. Keep it somewhere easy to find.
In Claude Desktop, open **Settings → Extensions → Install Extension**.
Select the downloaded `.mcpb` file.
Set the extension **Server URL** to your Liberator MCP Endpoint URL from the Liberator UI, for example:
```text theme={null}
https://api.cloudquant.ai/mcp
```
Once installed, Claude can use Liberator MCP tools such as `list_datasets`, `get_dataset_config`, and `query_dataset` from chat.
## Related
Connect ChatGPT with the MCP connector and OAuth
Add Liberator as an MCP tool in a Copilot Studio agent
# Microsoft Copilot Studio
Source: https://knowledge.cloudquant.com/integrations/copilot
Connect Microsoft Copilot Studio to CloudQuant Data Liberator with MCP and OAuth 2.0
# Connecting Liberator to Microsoft Copilot Studio
Add Liberator as an MCP connector in a **Copilot Studio** agent using **OAuth 2.0**, so each user signs in with their own Liberator credentials. Configure this from within Copilot Studio (not Microsoft 365 or Teams). An existing agent is required.
## Prerequisites
* Access to [Microsoft Copilot Studio](https://copilotstudio.microsoft.com/) with permission to edit an agent
* An existing Copilot Studio agent
* A CloudQuant Data Liberator account with valid username and password
* The Liberator **MCP Endpoint URL** from the Liberator UI (for example, `https://api.cloudquant.ai/mcp`)
Requires **Streamable HTTP** transport and **OAuth 2.0 Dynamic Client Registration (DCR)**. The connector can be shared org-wide, but each user creates their own connection by signing in with their own Liberator credentials. No shared API keys are needed.
## Setup steps
In your agent, go to **Tools → New Tool** and select **Model Context Protocol (MCP)**.
Enter a server name and description, and set **Server URL** to your Liberator MCP Endpoint URL, for example:
```text theme={null}
https://api.cloudquant.ai/mcp
```
Under **Authentication**, select **OAuth 2.0** with **Type** set to **Dynamic discovery**, then click **Create**.
On the **Add tool** screen, open the **Connection** dropdown, select **Create new connection**, then **Create**.
Sign in with your Liberator username and password and grant access (profile, `openid`, email).
Click **Add and configure**. Copilot auto-registers the Liberator tools (`list_datasets`, `get_dataset_config`, `query_dataset`).
## Using the connector
After the tool is added, Liberator MCP tools are available to the agent:
| Tool | Purpose |
| -------------------- | ------------------------------------------------- |
| `list_datasets` | List available Liberator datasets |
| `get_dataset_config` | Fetch configuration and schema for a dataset |
| `query_dataset` | Query a Liberator dataset by name with parameters |
Each user who uses the connector signs in with their own Liberator credentials, so entitlements stay per-user.
## Related
Connect ChatGPT with the MCP connector and OAuth
Install the Liberator .mcpb extension in Claude Desktop
# Grafana
Source: https://knowledge.cloudquant.com/integrations/grafana
Connect your Grafana instance to CloudQuant Data Liberator's Prometheus stack using a rotatable Bearer token
# Grafana integration
CloudQuant Data Liberator exposes a curated, read-only slice of its internal Prometheus stack at a Bearer-authenticated endpoint. You can wire your own Grafana (or any Prometheus-API-compatible client) to the same metrics that power the in-product **System Monitoring** view.
This integration is intended for ops teams, on-call rotations, and infrastructure dashboards that need to live alongside metrics from systems outside CloudQuant.
## At a glance
| | |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Endpoint** | `https:///metrics-api-bearer/` |
| **Authentication** | `Authorization: Bearer cqm_<48-hex-chars>` |
| **Protocol** | Standard Prometheus HTTP API (`/api/v1/query`, `/api/v1/query_range`, `/api/v1/series`, `/api/v1/labels`, `/api/v1/label//values`, `/api/v1/status/buildinfo`, `/federate`) |
| **Surface** | Read-only. Admin endpoints (`/api/v1/admin/*`) and remote-write are gated off at the Prometheus server, not just at the gateway. |
| **Token management** | Liberator UI → **System Monitoring → Grafana Integration** tab |
## Prerequisites
* A user with **super-admin** privileges on your Liberator instance (token management is a super-admin operation).
* Network reachability from your Grafana host to your Liberator hostname over HTTPS.
* Grafana 9.x or newer (any version that supports custom HTTP headers on the Prometheus datasource).
## Step 1 — Issue a Bearer token
1. Sign in to the Liberator UI as a super-admin.
2. Open **System Monitoring** from the top navigation.
3. Click the **Grafana Integration** action in the upper-right of the page.
4. Click **Generate token**. The new token is shown **exactly once**, so copy it immediately.
Tokens follow the format `cqm_` + 48 hex characters. Every token issued through this dialog is logged with the issuing user and timestamp; tokens can be listed and revoked from the same dialog.
The full token value is only visible at issue time. Treat it like a password: store it in your secrets manager and never commit it to source control. If a token is lost, revoke it and issue a new one rather than trying to recover it.
Issue one token per logical consumer (e.g. `grafana-prod`, `federated-prometheus-eu`) so you can revoke a single consumer's access without disrupting others.
## Step 2 — Configure the Grafana datasource
### Through the Grafana UI
1. **Connections → Data sources → Add data source → Prometheus.**
2. Set the fields below:
| Field | Value |
| ------------------------- | ------------------------------------------------------- |
| **Name** | `cloudquant-liberator` (or any label you prefer) |
| **Prometheus server URL** | `https:///metrics-api-bearer` |
| **HTTP method** | `POST` (recommended, handles longer queries) |
| **Scrape interval** | `30s` (matches the upstream Prometheus scrape interval) |
3. Under **Custom HTTP Headers**, click **+ Add header** and set:
* **Header** = `Authorization`
* **Value** = `Bearer cqm_`
4. Click **Save & test**. You should see **"Successfully queried the Prometheus API."**
### Through provisioning (recommended for production)
For reproducible setups, drop this file at `/etc/grafana/provisioning/datasources/cloudquant.yaml`:
```yaml theme={null}
apiVersion: 1
datasources:
- name: cloudquant-liberator
uid: cloudquant-liberator
type: prometheus
access: proxy
url: https:///metrics-api-bearer
isDefault: true
editable: true
jsonData:
httpMethod: POST
timeInterval: 30s
prometheusType: Prometheus
httpHeaderName1: Authorization
secureJsonData:
httpHeaderValue1: "Bearer cqm_"
```
Restart Grafana (or send `SIGHUP`) to pick up the file. The token lives in `secureJsonData`, which Grafana persists encrypted at rest.
## Step 3 — Run a query
Once the datasource is healthy, you can browse it in **Explore** or build dashboards against it. A few queries to verify end-to-end connectivity:
```promql theme={null}
# All scrape targets currently up
sum(up)
# Request rate by HTTP status class through the Liberator gateway
sum by (envoy_response_code_class) (rate(envoy_http_downstream_rq_xx[5m]))
# Liberator waiting-room active connections
liberator_waitingroom_connections_active
# p95 upstream request latency
histogram_quantile(
0.95,
sum by (le) (rate(envoy_cluster_external_upstream_rq_time_bucket[5m]))
)
```
## What's exposed
The Bearer endpoint forwards to the same Prometheus server that powers the in-product **System Monitoring** view, so anything you see there is also queryable here. Notable metric families:
| Prefix | What it covers |
| ----------------- | ---------------------------------------------------------------------------------- |
| `envoy_*` | Envoy Gateway — request rates, latency histograms, upstream connection health |
| `liberator_*` | Liberator application — waiting-room queue, in-flight query times, exporter health |
| `cache_manager_*` | Cache-manager worker pool status and error counters |
| `kube_*` | kube-state-metrics — pods, deployments, nodes, conditions |
| `container_*` | cAdvisor — per-container CPU, memory, filesystem, network |
| `node_*` | node-exporter — host CPU, memory, disk, network |
| `prometheus_*` | The Prometheus server itself — useful for sizing dashboards |
For a complete metric inventory in your environment, query `/api/v1/label/__name__/values` once you've connected.
## Rotating a token
1. Open the **Grafana Integration** dialog as in Step 1.
2. Issue a **new** token for the same logical consumer.
3. Update the new token in Grafana (UI: edit the datasource → swap the `Authorization` header value; provisioning: replace `httpHeaderValue1` and restart).
4. Wait until your Grafana datasource health check passes with the new token.
5. **Revoke** the old token from the same dialog.
This zero-downtime pattern is the same as the [AWS access-key rotation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey) workflow.
## Limitations
* **Read-only.** Prometheus admin endpoints (`/api/v1/admin/tsdb/*`) and the remote-write receiver are not exposed. Even tokened callers cannot mutate the TSDB through this endpoint.
* **One Prometheus.** The endpoint serves the Liberator cluster's primary Prometheus only. Federated views across multiple Liberator clusters require Prometheus-side federation on the consumer side; use `/metrics-api-bearer/federate` with appropriate `match[]` parameters.
* **CORS.** Browser-side direct calls are not supported (the gateway only allows server-side calls). Grafana proxies through its backend, so this restriction does not affect dashboard use.
## Troubleshooting
| Symptom | Diagnosis | Resolution |
| -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Save & test` returns *"non-JSON: `...`"* | The request is being redirected to the OIDC login page — the `Authorization` header isn't being sent, or the token is wrong. | Verify the header name is exactly `Authorization` and the value begins with `Bearer ` (with a trailing space) before the token. |
| `Save & test` returns *401 Unauthorized* | Token has been revoked, never issued, or has a typo. | Re-issue from the **Grafana Integration** dialog and update the datasource. |
| `Save & test` returns *403 Forbidden* | The Liberator gateway rejected the request before reaching Prometheus. | Check with your CloudQuant administrator that the bearer route is enabled on your cluster. |
| Health check passes but every query returns *"no data"* | Time range is outside Prometheus's retention window, or the metric name has changed in a recent upgrade. | Open **Explore**, hit `/api/v1/label/__name__/values` against the datasource, and confirm the metric exists. |
| Range queries fail with *"context deadline exceeded"* | Query window × resolution exceeds the gateway's response timeout. | Use the in-product **System Monitoring** view as a sanity check on the query, then widen your Grafana panel's `interval` or narrow the time range. |
# Integrations Overview
Source: https://knowledge.cloudquant.com/integrations/overview
Connect CloudQuant Data Liberator to AI assistants and observability tools
# Integrations
Connect Liberator to external AI assistants and observability tools. AI clients use the Liberator **MCP** endpoint; Grafana uses the Bearer-authenticated metrics API.
## AI assistants (MCP)
Available in Liberator **2.3** and later. Point your client at the Liberator MCP Endpoint URL from the Liberator UI (for example, `https://api.cloudquant.ai/mcp`), then follow the setup guide for your assistant.
MCP connector with OAuth in ChatGPT Apps
MCP tool with OAuth 2.0 Dynamic discovery
Install the Liberator .mcpb extension bundle
## Observability
Available in Liberator **2.2** and later.
Prometheus metrics via a rotatable Bearer token
## Related
Configure Liberator data source connections and datasets
In-product cluster, queue, dataset, and usage views
# S3 Bucket Setup
Source: https://knowledge.cloudquant.com/integrations/s3-bucket-setup
How to set up your S3 bucket so that CloudQuant Data Liberator can connect and read data from it
# S3 bucket setup for CloudQuant Data Liberator
This guide walks you through setting up your Amazon S3 bucket for use with the CloudQuant Data Liberator platform.
## Prerequisites
* An AWS account with S3 access
* IAM permissions to create buckets and manage policies
* Your CloudQuant account credentials
## Step 1: Create an S3 bucket
1. Navigate to the [AWS S3 Console](https://s3.console.aws.amazon.com/)
2. Select **Create bucket**
3. Select a unique bucket name and region
4. Configure bucket settings as needed
## Step 2: Configure bucket policy
Add a bucket policy that grants CloudQuant Data Liberator read access:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::CLOUDQUANT_ACCOUNT_ID:root"
},
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"
]
}
]
}
```
Contact CloudQuant support for the correct AWS Account ID to use in the bucket policy.
## Step 3: Configure CORS (if needed)
If accessing data from web applications, configure CORS:
```json theme={null}
[
{
"AllowedHeaders": ["*"],
"AllowedMethods": ["GET"],
"AllowedOrigins": ["*"],
"ExposeHeaders": []
}
]
```
## Step 4: Generate access keys
1. Navigate to IAM in the AWS Console
2. Create a new IAM user or use an existing one
3. Generate an Access Key ID and Secret Access Key
4. Provide these credentials to CloudQuant securely
## Step 5: Verify connectivity
Contact CloudQuant support to verify that CloudQuant Data Liberator can successfully connect to your S3 bucket.
## Troubleshooting
| Issue | Solution |
| ---------------- | ---------------------------------------------- |
| Access Denied | Verify bucket policy and IAM permissions |
| Bucket Not Found | Confirm bucket name and region |
| Timeout | Check network connectivity and security groups |
# Introduction
Source: https://knowledge.cloudquant.com/introduction
Welcome to the CloudQuant Documentation - your comprehensive guide to the CloudQuant Data Liberator data platform
# Welcome to CloudQuant
CloudQuant provides institutional-grade financial data infrastructure through the **CloudQuant Data Liberator** platform — a simple point-in-time data access API for live or historical time series data.
## What is CloudQuant Data Liberator?
CloudQuant Data Liberator is CloudQuant's data delivery platform that provides:
* **Multi-language API support** — Python, C#, JavaScript, Java, R, RESTful, C++, and Excel
* **Time series data access** — Query historical and live financial datasets
* **70+ connectors** — so you don't have to move your data to get insights
* **Cloud-native integrations** — S3, Azure Blob Storage, and direct connectivity
## Quick start
Get started with CloudQuant Data Liberator for Python — the most popular way to access data
Explore the full API reference for all supported languages
Browse 70+ available datasets and data integrations
Configure connections to databases, cloud storage, and file systems
## Platform components
| Component | Description |
| --------------------------------- | -------------------------------------------------- |
| **CloudQuant Data Liberator API** | Core data access API with multi-language support |
| **Data Catalog** | 70+ integrated financial and alternative datasets |
| **Integrations** | Cloud platform connectivity (S3, Azure) |
| **Datasource Config** | Connect databases, cloud storage, and file systems |
## Need help?
* Visit the [CloudQuant Support Portal](https://www.cloudquant.com/tickets) for direct assistance
* Browse the documentation sections using the sidebar navigation
# Network Configuration
Source: https://knowledge.cloudquant.com/network-configuration
Configure your network environment for CloudQuant Data Liberator access
# Network configuration
This guide covers the network setup required to connect to the CloudQuant Data Liberator platform.
## Firewall requirements
Ensure the following outbound connections are permitted:
| Protocol | Port | Destination | Purpose |
| -------- | ------- | --------------------------------------------- | ------------------------------- |
| HTTPS | 443 | CloudQuant API endpoints | Data queries and authentication |
| TCP | Various | CloudQuant Data Liberator streaming endpoints | Live data feeds |
## Proxy configuration
If your organization uses a proxy server, configure the CloudQuant Data Liberator client accordingly:
```python Python theme={null}
import os
os.environ['HTTPS_PROXY'] = 'http://your-proxy:8080'
```
```bash Shell theme={null}
export HTTPS_PROXY=http://your-proxy:8080
export NO_PROXY=localhost,127.0.0.1
```
## DNS requirements
Ensure your DNS can resolve CloudQuant's service endpoints. Contact CloudQuant support for the current list of required domains.
## Troubleshooting
If you experience connectivity issues:
1. Verify outbound HTTPS (port 443) is open
2. Check proxy settings if applicable
3. Ensure DNS resolution is working
4. Contact [CloudQuant Support](https://www.cloudquant.com/tickets) for endpoint-specific assistance
# CloudQuant Data Liberator Overview
Source: https://knowledge.cloudquant.com/overview
Understanding the CloudQuant Data Liberator data platform architecture and capabilities
# CloudQuant Data Liberator overview
CloudQuant Data Liberator is CloudQuant's core data delivery platform that provides simple point-in-time data access via API for both live and historical time series data.
## Architecture
CloudQuant Data Liberator serves as the central hub for accessing CloudQuant's data ecosystem:
* **Data Ingestion** — Connects to 70+ data providers and integrates their feeds
* **Data Storage** — Manages time series data with efficient compression and indexing
* **Data Delivery** — Provides API access in multiple programming languages
* **Access Control** — Manages dataset entitlements and user permissions
## Supported languages
CloudQuant Data Liberator provides native client libraries for:
* **Python** — Full-featured client with Pandas DataFrame support
* **C#** — .NET client library
* **JavaScript** — Node.js module
* **Java** — Java client library
* **R** — R language client
* **RESTful** — HTTP REST API for any language
* **C++** — High-performance native client with Apache Arrow support
* **Excel** — Plug-in for direct spreadsheet access
## Key concepts
### Datasets
A dataset is a named collection of time series data. Use `liberator.datasets()` (Python) or equivalent methods to discover available datasets.
### Queries
Queries retrieve data from datasets using parameters like time range (`as_of`, `back_to`), symbols, and column selection.
### Timestamps and MUTS
CloudQuant Data Liberator uses microsecond timestamps (MUTS) for precise time series data alignment.
## Getting started
Choose your preferred language from the [API Reference](/api-reference/concepts/queries-large-datasets) section, or follow the [Python Getting Started Guide](/python-guide/getting-started) for the most common path.
# Understanding as_of and back_to
Source: https://knowledge.cloudquant.com/python-guide/as-of-and-back-to
Learn how the as_of and back_to parameters work with point-in-time datasets in CloudQuant Data Liberator.
# Understanding as\_of and back\_to
The `as_of` and `back_to` parameters define the start and end points for data retrieval. Rather than using "start" and "end," these terms accommodate datasets that are **point-in-time** — meaning different data may be returned based on the `as_of` date.
## What are point-in-time datasets?
Point-in-time datasets reflect data as it existed at a specific historical moment. A common example is revised earnings reports: if your `as_of` date precedes the revision date, you receive original values; if it matches or follows the revision, you get updated figures.
Machine learning datasets increasingly use point-in-time logic. Providers may retrain models and recalculate historical values. You typically want access to the original data from your trading period, and you adopt modified data only when you choose to.
Any point-in-time datasets will be labeled clearly as such in the dataset description in `liberator.datasets()`.
## Source of point-in-time data problems
Standard databases prioritize current information, which introduces **survivorship bias** into quantitative analysis. For instance:
* **SEC filings** — Historical earnings may have been restated due to accounting errors, regulatory scrutiny, or fraud. You need the original reported numbers alongside modification timestamps.
* **Index constituents** — The S\&P 500's composition changes over time. Analysis based on today's index membership creates survivorship bias. For rigorous analysis, you need exact historical index compositions and all changes.
Point-in-time data discipline prevents forward-looking bias that compromises quantitative research integrity.
# Usage of the as_of Parameter
Source: https://knowledge.cloudquant.com/python-guide/as-of-parameter
How to use the as_of parameter to retrieve data as it was known at a particular point in time.
# Usage of the as\_of parameter
The `as_of` parameter allows you to query historical data from a specific moment. You can specify it as a date alone (e.g., `'2024-07-08'`) which CloudQuant Data Liberator interprets as midnight that date, or with specific times down to nanosecond precision.
All CloudQuant Data Liberator times are New York Time (US ET).
## Default behavior
When you omit the `as_of` parameter, CloudQuant Data Liberator assumes the current date and time.
In Liberator **2.4** and later, a future `as_of` on a live-appending dataset is rejected or bounded instead of holding a query worker in an empty-fetch loop. Use `as_of` at or before "now" unless you are intentionally querying forward-dated records that already exist in the source.
## Example query
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-10',
back_to = '2024-07-10',
symbols = ['GOOG', 'AAPL', 'MSFT']))
```
This returns data for the specified date range.
## Without back\_to parameter
If you specify `as_of` but omit or set `back_to` to `None`, the query returns data through yesterday's completed trading bar:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-18',
symbols = ['GOOG', 'AAPL', 'MSFT']))
```
## Jupyter notebook configuration
To display more rows in Jupyter:
```python theme={null}
import pandas as pd
pd.set_option('display.max_rows', 400)
```
Consider your data volume and test with smaller queries initially. See [Query Considerations](/python-guide/query-considerations) for more details.
# Usage of the back_to Parameter
Source: https://knowledge.cloudquant.com/python-guide/back-to-parameter
How to use the back_to parameter to define the start of a time range in CloudQuant Data Liberator queries.
# Usage of the back\_to parameter
The `back_to` parameter is a datetime parameter in CloudQuant Data Liberator queries that defines the start point of a time range for data retrieval.
## Parameter format
You can specify `back_to` in two ways:
**Date only:**
```
'2024-07-08'
```
CloudQuant Data Liberator interprets this as midnight: `'2024-07-08 00:00:00.000000'`
**Specific time:**
You can set it to any time down to nanosecond precision.
## Important notes
All CloudQuant Data Liberator times are New York Time (US ET).
Start with small queries to understand data amounts before requesting large date ranges.
* If `back_to` is omitted or set to `None`, CloudQuant Data Liberator returns the most recent value at the specified `as_of` time.
## Default behavior example
Without `back_to`, the query returns the latest available data:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2023-12-25',
symbols = ['GOOG', 'AAPL', 'MSFT']
))
```
Since December 25, 2023 was not a trading day, CloudQuant Data Liberator returned data from the most recent trading day (December 22, 2023).
# Batch Downloading Data
Source: https://knowledge.cloudquant.com/python-guide/batch-downloading
How to efficiently download large datasets from CloudQuant Data Liberator using batch processing and chunked downloads.
# Batch downloading data
Batch downloading is especially useful when you need to download extremely large datasets. This approach offers several advantages over traditional query methods.
## Key advantages
* **Streamed delivery** — Data arrives as a stream rather than waiting for complete server-side processing
* **Resumable downloads** — Write data to file as it arrives, allowing restart points if connection interrupts
* **Memory efficiency** — Avoids double memory usage common with standard DataFrame downloads
* **Simplicity** — Code length is comparable to standard query-to-DataFrame approaches
## Small batch download example
The simplest implementation submits a single large query and writes batches to file:
```python theme={null}
import liberator, time
start_time = time.time()
for batch in liberator.query(name = 'minute_bars', symbols = None,
as_of = '2024-07-01', back_to = '2024-01-01'):
batch.to_pandas().to_csv("minute_bar_data.csv", mode='a')
print("The query + saving took", (time.time() - start_time)/60.0, " minutes to run")
```
An enhanced version prevents empty files and handles headers correctly:
```python theme={null}
import liberator, time
start_time = time.time()
for i, batch in enumerate(liberator.query(name='daily_bars', as_of = '2024-07-01',
back_to = '2024-01-01', symbols = None)):
if not len(batch):
continue
batch.to_pandas().to_csv('daily_bars.csv', mode='a' if i else 'w',
header=False if i else True)
print("The query + saving took", (time.time() - start_time)/60.0, " minutes to run")
```
## Large batch download process
For very large datasets, splitting downloads into monthly chunks prevents resource constraints:
```python theme={null}
# Define the start and end dates and other setup info for the loop
start_year = 2018
start_month = 3
end_year = 2024
end_month = 06
symbols = None
header = True
individual = 0
header_in_individual = 0
dataset = 'minute_bars'
import liberator, time
back_to_year = start_year
back_to_month = start_month
old_month = None
while (back_to_year < end_year) or (back_to_year == end_year and back_to_month <= end_month):
if back_to_month == 12:
as_of_month = 1
as_of_year = back_to_year + 1
else:
as_of_month = back_to_month + 1
as_of_year = back_to_year
back_to = f"{back_to_year:04d}-{back_to_month:02d}-01"
as_of = f"{as_of_year:04d}-{as_of_month:02d}-01"
print(f"Downloading {dataset} {back_to} to {as_of}", end='')
start_time = time.time()
head = header if (old_month==None) else False
for batch in liberator.query(name=dataset, as_of = as_of,
back_to = back_to, symbols = symbols):
if not len(batch):
continue
if individual:
batch.to_pandas().to_csv(dataset+'_'+back_to+'_'+as_of+'.csv',
mode='a', header=header_in_individual or head)
else:
batch.to_pandas().to_csv(dataset+'_all.csv', mode='a', header=head)
print(" Query + save took", (time.time() - start_time)/60.0, " minutes to run. head:",head)
back_to_year = as_of_year
back_to_month = as_of_month
old_month = as_of_month
```
This monthly chunking approach allows downloading even the largest datasets without memory constraints, making it ideal for production data pipelines.
# Selecting Specific Columns
Source: https://knowledge.cloudquant.com/python-guide/column-selection
How to use the fields parameter to restrict which columns are returned in CloudQuant Data Liberator query results.
# Selecting specific columns
CloudQuant Data Liberator automatically returns a minimum default set of columns for each dataset, which varies but typically includes `_seq`, `muts`, `timestamp`, and `symbol`. Since some datasets contain hundreds of columns, you may want to limit results to specific fields, especially on slower connections.
## Filtering columns with the fields parameter
To reduce returned columns, pass the `fields` parameter with your desired column list:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-22',
symbols = ['AAPL', 'GOOGL'],
fields = ['Open', 'Close']
))
```
CloudQuant Data Liberator places your selected columns at the front of the DataFrame, followed by the default columns.
## Discovering available columns
To identify column names in a dataset, use one of these approaches:
### Using the schema function
```python theme={null}
liberator.datasets(schema=True)['nameOfDataset']
```
### Inspecting a sample query result
```python theme={null}
df.columns
list(df.columns)
print(list(df.columns))
```
Example output for `daily_bars`:
```python theme={null}
['_seq', '_dsname', 'timestamp', 'msg_len', 'msg', 'muts', 'symbol',
'length', 'open', 'high', 'low', 'close', 'volume', 'vwap', 'bvwap',
'spread', 'bidvol', 'askvol', 'count', 'avgdelta', 'Date', 'Time',
'Hour', 'DateTime', 'DateHour']
```
# Checking Dataset Access
Source: https://knowledge.cloudquant.com/python-guide/dataset-access
How to discover which datasets you have access to and view their schemas and descriptions.
# Checking dataset access
CloudQuant Data Liberator provides built-in commands to check your dataset permissions, view available datasets, and access detailed schema information.
## Commands
### Browse the catalog
```python theme={null}
liberator.datasets()
```
Returns datasets visible in the catalog for your account — public datasets and those you are entitled to. Private datasets you are not entitled to are hidden. See [Dataset visibility](/administration/dataset-visibility).
### View entitled datasets only
```python theme={null}
liberator.datasets(entitled=True)
```
Returns datasets you are individually entitled to (in addition to public datasets, depending on client defaults).
### Get dataset details
```python theme={null}
liberator.datasets(details=True)('daily_bars')
```
Retrieves a detailed description of a specific dataset. Replace `'daily_bars'` with your target dataset name.
In Liberator **2.4** and later, details also include catalog-card health: date range covered, symbol count, status (for example, Ready), and last updated. Administrators can configure how often symbol discovery refreshes so new keys appear here after the source changes.
Super Admins can also query **`usage_log`**, a private per-query usage dataset that does not appear in the standard catalog. See [Query usage log](/administration/usage-log).
### View dataset schema
```python theme={null}
liberator.datasets(schema=True)('daily_bars')
```
Returns the complete data dictionary and schema for a specific dataset. Replace `'daily_bars'` as needed.
### Fetch concordance metadata (2.3+)
```python theme={null}
liberator.metadata(name='daily_bars', summary=True, symbol=False)
```
Returns concordance-backed row counts, symbol counts, and start/end timestamps without downloading source rows. See [Dataset metadata](/api-reference/concepts/dataset-metadata) for the `summary` × `symbol` grain matrix.
Use these commands to explore available data before writing queries. The schema information is especially useful for identifying column names to use with the [fields parameter](/python-guide/column-selection).
# Getting Started with CloudQuant Data Liberator for Python
Source: https://knowledge.cloudquant.com/python-guide/getting-started
Learn how to set up and configure the CloudQuant Data Liberator Python library to connect to CloudQuant's data platform.
# Getting started with CloudQuant Data Liberator for Python
This guide introduces the CloudQuant Data Liberator library, available across multiple programming languages via the CloudQuant API platform.
## Basic setup
### Quick start
For environments where CloudQuant Data Liberator is pre-installed, simply use:
```python theme={null}
import liberator
```
### Installation steps
If adding CloudQuant Data Liberator to your own Python environment (local installation, Jupyter, or IDE):
1. Download the required files from your Liberator Profile page
2. Install dependencies:
```bash Conda theme={null}
conda create -n liberator python=3.7
conda activate liberator
conda install -c anaconda requests pyopenssl cryptography six pytz
conda install pandas
conda install -c conda-forge httpx async_generator pyarrow=8.0.0
```
```bash pip theme={null}
python -m venv liberator
source liberator/bin/activate # Windows: liberator\Scripts\activate
pip install requests pandas httpx async_generator "pyarrow==8.0.0" pyOpenSSL cryptography six pytz
```
3. Place files in either:
* The same folder as your code, **or**
* A central Python location (`liberator.py` only — see note below)
Putting `liberator.py` in `site-packages` lets you `import liberator` from anywhere, but `liberator.json` is still read from your working directory unless you set `liberator.auth = '/path/to/liberator.json'`.
To find your Python's site-packages directory:
```python theme={null}
import site
print(site.getsitepackages())
```
### Required files
| File | Purpose |
| ---------------- | -------------------------------------------------------------- |
| `liberator.py` | The Python library |
| `liberator.json` | Pre-populated credentials (username and token) |
| `liberator.pfx` | Security certificate (only required for releases prior to 2.0) |
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx`.
## Server configuration
Files downloaded from your Liberator Profile page include the correct server URL preconfigured in the client.
### Override server URL
To connect to a different instance, set the URL explicitly:
```python theme={null}
liberator.url = 'https://mywidgets.cloudquant.ai/'
liberator.url = 'http://10.10.56.127:47753/'
```
Use `https://` for domain names and `http://` for direct IP addresses. Specify port numbers as shown above when required.
Once your client is configured, you are ready to begin pulling data.
# Accessing Data from a Live Trading Environment
Source: https://knowledge.cloudquant.com/python-guide/live-trading-environment
How to use CloudQuant Data Liberator data in live algorithmic trading environments, including data transfer strategies and live streaming.
# Accessing data from a live trading environment
This guide addresses how to use CloudQuant Data Liberator data in live algorithmic trading environments, particularly when third-party APIs cannot be integrated directly.
## Key challenges
Some organizations restrict third-party APIs in live trading systems. When this applies to you, retrieve the data locally and transfer it to the trading server through approved methods.
## Data transfer solutions
### Low frequency approaches
For infrequently updated data, set up shared file storage accessible to both the live trading application and your local servers. Push data to these locations on a schedule, and coordinate with your Operations department to arrange access.
### High frequency solutions
For more frequent intraday updates, messaging services like [ZeroMQ](https://zeromq.org/) are recommended as lightweight alternatives. ZeroMQ operates peer-to-peer without requiring intermediate broker infrastructure.
## Accessing live data from CloudQuant Data Liberator
CloudQuant Data Liberator delivers live datasets with similar simplicity to historical data retrieval. The platform supports seamless transitions between historical and real-time data.
### Example code
```python theme={null}
import liberator
liberator.url = 'https://weather.cloudquant.ai/'
yesterday = '2024-08-01'
res = liberator.query(name = 'us_live_weather', symbols = '78758', as_of = 'live', back_to = yesterday)
for batch in res:
df = batch.to_pandas()
print(df)
```
When `back_to` is specified, the query delivers historical data from that point forward, then automatically transitions to live streaming. Omitting `back_to` initiates live data immediately at the vendor's defined frequency.
# Pulling Data into a Pandas DataFrame
Source: https://knowledge.cloudquant.com/python-guide/pandas-dataframe
How to extract data from CloudQuant Data Liberator datasets into Python pandas DataFrames with practical examples.
# Pulling data into a Pandas DataFrame
This guide demonstrates how to extract data from CloudQuant Data Liberator datasets into Python pandas DataFrames. The examples use `daily_bars`, a dataset containing US Equity daily OHLCV (Open, High, Low, Close, Volume) data with additional fields.
## Example: pulling daily bars for AAPL
### Multi-line approach
```python theme={null}
today = '2024-06-20'
oneMonthAgo = '2024-05-20'
dataset = 'daily_bars'
mySymbols = 'AAPL'
query = liberator.query(name=dataset, symbols=mySymbols, as_of=today, back_to=oneMonthAgo)
df = liberator.get_dataframe(query)
```
### Single-line approach
```python theme={null}
df = liberator.get_dataframe(liberator.query(name='daily_bars', symbols='AAPL', as_of='2024-06-20', back_to='2024-05-20'))
```
## Key features
* **Flexible symbol input** — accepts single symbols, lists, or `None` (all symbols)
* **Granular time selection** — supports daily, minute, second, and nanosecond resolution
* **Simple syntax** — collapsible into single-line queries for efficiency
The query returns a properly formatted pandas DataFrame containing the requested historical data.
# Considerations When Submitting a Query
Source: https://knowledge.cloudquant.com/python-guide/query-considerations
Important constraints, timestamp conventions, and best practices to understand before querying CloudQuant Data Liberator datasets.
# Considerations when submitting a query
Before submitting your first CloudQuant Data Liberator query, there are several important factors to keep in mind regarding constraints, timestamps, and data volume.
## Constraints on a query
The system imposes only two limitations: the **time range** selected and the **symbols** chosen for analysis.
All timestamps in CloudQuant Data Liberator use **New York Time (US ET)**, regardless of the market's geographic location. This standardization helps you understand when data becomes available to you.
## Notes on timestamps
Datasets frequently contain multiple timestamps serving different purposes:
* The **primary timestamp** indicates when you can access the data.
* **Secondary timestamps** may represent exchange or execution times, which typically occur before or coincide with the main timestamp.
For calculated datasets like minute bars, there is often a `bar_time` distinct from the availability timestamp. In US equities markets, the first daily bar concludes at `9:30:59.999999`, with a `bar_time` of `9:31:00.000000`.
Calculated datasets may show vendor timestamps differing from CloudQuant's timestamp. CloudQuant's timestamp is designed to give you the best possible idea of the **actual available time of the data**.
## Think before you query
Before submitting queries, assess the likely data volume.
A simple request for AAPL daily bars for April might return approximately 30 rows. The same parameters applied to NBBO data (all bids and asks) could generate enormous datasets requiring extensive system processing.
Best practice involves querying unfamiliar datasets with **limited date ranges initially**, then progressively expanding timeframes as you become comfortable with the data scale.
In Liberator **2.4** and later, administrators can also set a **maximum number of days** a single query may span on a dataset. If you hit that cap, the error message names the limit and the configured value — narrow `back_to` / `as_of` and retry. See [Fine-grained permissions](/administration/fine-grained-permissions#limits-and-settings).
For guidance, contact [customer\_success@cloudquant.com](mailto:customer_success@cloudquant.com).
# Query Parameter Combinations
Source: https://knowledge.cloudquant.com/python-guide/query-parameters
A summary of all CloudQuant Data Liberator query parameters and how different combinations affect the results you receive.
# Query parameter combinations
This guide explains the main parameters for CloudQuant Data Liberator queries and the expected results from different combinations.
## Core parameters
| Parameter | Description | Default |
| --------- | ------------------------------------------------------------------------------------------ | -------------------- |
| `name` | Dataset identifier (required) | -- |
| `symbols` | Single symbol, list, or (2.4+) `LIKE` pattern with `%`, e.g. `['AAPL','GOOG']` or `'PWR%'` | `None` (all symbols) |
| `back_to` | Query start date | `None` |
| `as_of` | Query end date | Current date/time |
| `fields` | Specify desired columns to reduce download time | All columns |
| `stats` | Set to `'Total'` for symbol counts instead of full results | -- |
| `where` | Custom SQL predicate appended to the generated filters | `None` |
| `sql` | Full raw SQL query (replaces `name`, time range, and other construction params) | `None` |
Use `liberator.datasets(entitled=True)` to list all datasets available to you.
See the [Query Parameters Reference](/api-reference/concepts/query-parameters) for the full parameter list, including superquery, N-Query, and authentication options.
## Query examples
### All parameters included
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-16',
symbols = ['AAPL', 'GOOGL']))
```
Returns data for specified symbols within the date range.
### With a symbol pattern (2.4+)
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'energy_contracts',
as_of = '2026-08-02',
back_to = '2026-08-01',
symbols = 'PWR%'))
```
An unescaped `%` is a `LIKE` wildcard, so this returns every key that starts with `PWR`. Mix exact keys and patterns in one list (`['AAPL', 'PWR%']`). Escape a literal percent as `\%`. See [Pattern filters](/python-guide/symbols-and-key-fields#pattern-filters-24).
### Without symbols
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-16'))
```
Defaults to all available symbols.
### Without `as_of`
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
back_to = '2024-07-22'))
```
Defaults `as_of` to current time, returning data from present back to specified date.
### Without `back_to`
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2023-12-25'))
```
Returns the most recent data point per symbol as of the specified date. Look-back distance varies by dataset density.
### Minimal query (name only)
```python theme={null}
df = liberator.get_dataframe(liberator.query(name = 'daily_bars'))
```
Returns the latest data point for all symbols in the dataset.
## Custom `where` filters
Use `where` with a standard parameter-based query to append extra SQL predicates on top of the filters CloudQuant Data Liberator generates for time range and symbols. The clause is combined with `AND` — do not include the `WHERE` keyword.
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name='daily_bars',
as_of='2024-07-24',
back_to='2024-07-16',
symbols=['AAPL', 'GOOGL'],
where='volume > 1000000'
))
```
`where` works with time-range, LKV, and N-Query modes. Column names that match the dataset schema are automatically quoted when needed.
### Multi-dataset / concordance-style filters
For datasets keyed by more than symbol (for example concordance tables with a `dataset` column), use `where` to narrow the result set:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name='concordance',
as_of='2025-01-01',
back_to='2024-01-01',
where="dataset = 'my_derived_dataset'"
))
```
`where` cannot be combined with `sql`. Use one approach or the other.
## Raw `sql` queries
Pass a complete SQL `SELECT` statement via `sql` instead of `name`, `back_to`, `as_of`, and related parameters. When `sql` is present, parameter-based query construction is skipped.
```python theme={null}
df = liberator.get_dataframe(liberator.query(
sql='''
SELECT *
FROM "daily_bars"
WHERE muts >= 1704067200000000
AND muts < 1704153600000000
AND symbol = $$AAPL$$
''',
user='my_user',
system='API'
))
```
### SQL requirements
* The statement must be a **`SELECT`** (including `UNION` / `INTERSECT` / `EXCEPT` branches).
* Every `SELECT` arm must include an explicit **`WHERE`** clause.
* By default, the `WHERE` clause must include a **time-range predicate** on `muts` or another configured column (`=`, `>`, `>=`, `<`, `<=`, or `BETWEEN`). Use microsecond timestamps for `muts` values.
* Quote dataset names with double quotes when they contain special characters.
* Use `$$symbol$$` dollar-quoting for string literals in filters.
### Parameters allowed with `sql`
When using `sql`, only connection, auth, and transfer options are permitted alongside the statement — for example `user`, `system`, `token`, `compress`, `json_xfer`, and `batch_size`. Supplying non-empty values for `name`, `back_to`, `as_of`, `where`, or other construction parameters raises an error.
Raw SQL bypasses the convenience of `name` / `symbols` / `back_to` / `as_of`. Prefer parameter-based queries unless you need full control over the generated SQL.
# Required Timestamps in Queries
Source: https://knowledge.cloudquant.com/python-guide/required-timestamps
Every Liberator query must include a valid date or time range using back_to and as_of (or an explicit WHERE on time columns)
# Required timestamps in queries
Liberator enforces a **time-based predicate on all queries**. Every query must include a valid, non-zero date or time range. Queries without one return an error and no data.
Previously, a query with no time parameters could return an entire dataset. That behavior was removed to prevent unexpectedly large result sets and unnecessary load.
Review saved notebooks, scripts, and scheduled jobs. Any query that omits `back_to` / `as_of` (or an explicit time `WHERE` in raw `sql`) will fail until updated.
Applies to Liberator **2.1 and later**.
## Valid timestamps
A valid time scope must:
* Cover a **specific date, date range, or datetime range** — not zero, null, or empty.
* Use **`back_to` and `as_of`** for parameter-based queries (Liberator's point-in-time window). See [Understanding as\_of and back\_to](/python-guide/as-of-and-back-to).
* Fall within the dataset's available coverage (valid dates outside coverage return an **empty** result, not a timestamp error).
### Correct examples
```python theme={null}
import liberator
# Single day
df = liberator.get_dataframe(liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2026-01-01",
as_of="2026-01-01",
))
# Date range
df = liberator.get_dataframe(liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2026-01-01",
as_of="2026-06-30",
))
# Datetime range
df = liberator.get_dataframe(liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2026-06-01 09:30:00",
as_of="2026-06-01 16:00:00",
))
```
### Incorrect examples
```python theme={null}
# Missing back_to and as_of — error
liberator.query(name="my_dataset", symbols=["AAPL"])
# Null dates — error
liberator.query(name="my_dataset", symbols=["AAPL"], back_to=None, as_of=None)
```
### Raw `sql` queries
Every `SELECT` must include an explicit **`WHERE`** clause with a time-range predicate on `muts` or another configured time column. See [Query parameters — Raw SQL](/python-guide/query-parameters#raw-sql-queries).
## What the error looks like
The API or client returns an error indicating a time-based predicate is required. This is expected enforcement, not a platform failure.
## Update existing queries
1. **Find affected calls** — Search for `liberator.query()` without `back_to` / `as_of`, or raw `sql` without a time `WHERE`.
2. **Choose the right window** — Daily jobs: parameterize to yesterday or today. Historical pulls: set explicit start and end dates.
3. **Add parameters** — Use the examples above.
4. **Test** — Run manually before re-enabling schedules.
## FAQ
**I used to pull a full dataset for reference. What now?**\
Set `back_to` and `as_of` to the dataset's available range (ask your administrator or check dataset details in the admin portal).
**Error but I provided dates?**\
Confirm values are not null/empty, `back_to` is not after `as_of`, and dates use `YYYY-MM-DD` or `YYYY-MM-DD HH:MM:SS`.
**Valid dates but empty results?**\
The timestamp is accepted; the range may fall outside dataset coverage.
**AI assistant queries?**\
Assistants must specify a time range. Timeless requests should prompt for dates; otherwise the query fails like any other.
## Related
Point-in-time semantics for Liberator datasets
Full parameter reference including raw SQL rules
# Symbols and Key Fields
Source: https://knowledge.cloudquant.com/python-guide/symbols-and-key-fields
Understanding the symbol parameter in CloudQuant Data Liberator queries, how to discover available symbols, and how to query for all symbols in a dataset.
# Symbols and key fields
The **symbol** parameter serves as the key for querying datasets in CloudQuant Data Liberator. The platform initially focused on US Equity Symbols, but its datasets have expanded to include diverse data types such as shipping, weather, and market research. These datasets use varied keys like zip codes and geographic locations. CloudQuant Data Liberator keeps the `symbols` parameter name for consistency with existing integrations.
**Default Value:** `None` (returns all symbols/keys, including blanks/NaN values)
`symbols` and `keys` are synonyms — use either name.
When setting your query parameters, consider the amount of data you may be requesting and start small until you understand data volume patterns for each symbol/date combination.
## Determining unique symbols in a dataset
The most straightforward approach is to query and count the data. Here is an example using the `daily_bars` dataset:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-22',
symbols = None))
df
```
This returns over 20,000 rows. To count unique symbols:
```python theme={null}
len(df.symbol.unique())
# Output: 10524
```
## Storing and displaying symbols
To store symbols in a variable:
```python theme={null}
allsym = list(df.symbol.unique())
```
To display the complete list (with caution — lists can be lengthy):
```python theme={null}
print(list(df.symbol.unique()))
```
## Query methods for all symbols
Either explicitly set `symbols = None` or omit the parameter entirely, as it defaults to `None`.
## Pattern filters (2.4+)
In Liberator **2.4** and later, `symbols` / `keys` accept SQL `LIKE` patterns. An unescaped `%` matches any sequence of characters, so you can request a product family or naming pattern without listing every contract.
```python theme={null}
# All keys that start with PWR
df = liberator.get_dataframe(liberator.query(
name='energy_contracts',
symbols='PWR%',
back_to='2026-08-01',
as_of='2026-08-02',
))
# Mix exact keys and patterns
df = liberator.get_dataframe(liberator.query(
name='energy_contracts',
symbols=['AAPL', 'MSFT', 'PWR%'],
back_to='2026-08-01',
as_of='2026-08-02',
))
```
| Pattern | Matches |
| ------- | --------------------------------------------- |
| `PWR%` | `PWR`, `PWR-H`, `PWR_CAL_26` |
| `G%GL` | `GOGL`, `GAGL`, `GOOGLE` |
| `A\%PL` | only the literal key `A%PL` (escaped percent) |
Exact keys (no unescaped `%`) still use equality, including lists of tickers. Liberator only switches to `LIKE` for values that contain `%`.
Start with a short date range when you first use a pattern. A leading `%` on a large dataset can return far more rows than an exact-symbol query.
See the [query parameters reference](/api-reference/concepts/query-parameters#dataset--symbol-parameters) for `symbols` and `keys`.
# Security & Access
Source: https://knowledge.cloudquant.com/security-and-access
CloudQuant Data Liberator security model, authentication, and access control
# CloudQuant Data Liberator security and access
CloudQuant Data Liberator provides enterprise-grade security for data access and distribution.
## Authentication
Access to CloudQuant Data Liberator requires valid credentials provided by CloudQuant. Authentication is handled through:
* **API Keys** — Used for programmatic access via client libraries
* **User Credentials** — Used for interactive access via Excel and CQAI
## Dataset entitlements
Access to specific datasets is controlled through entitlements:
* Each user or organization is granted access to specific datasets
* Use `liberator.datasets()` (Python) or equivalent to see your available datasets
* Organization administrators can manage users and entitlements from the admin portal — see [Administration Overview](/administration/overview)
* Contact CloudQuant support to request access to additional datasets
### For administrators
Add, create, deactivate users and grant or revoke dataset access
Restrict query date ranges, row limits, keys, and columns
## Network security
CloudQuant Data Liberator supports secure connectivity through:
* **TLS/SSL encryption** for all data in transit
* **IP whitelisting** for restricted access environments
* **VPN connectivity** for on-premises deployments
## Anti-virus considerations
When running CloudQuant Data Liberator client libraries locally, ensure your anti-virus software does not block:
* CloudQuant Data Liberator client executables
* Network connections to CloudQuant data endpoints
* Local cache directories used by the client
See the [Network Configuration](/network-configuration) page for detailed setup instructions.
# System Monitoring Overview
Source: https://knowledge.cloudquant.com/system-monitoring/overview
Built-in observability for the CloudQuant Data Liberator stack, with a tour of each tab and when to look at it
# System monitoring
The **System Monitoring** page in the Liberator UI surfaces the most operationally relevant signals from the Liberator stack in a single place: cluster health, queue depth, dataset access patterns, long-running queries, and license usage.
It's intended for super-admins, on-call engineers, and capacity planners. End users do not see this page.
The System Monitoring page is backed by the same Prometheus instance you can connect Grafana to. See the [Grafana](/integrations/grafana) integration guide if you want the same data alongside metrics from systems outside CloudQuant.
## Opening system monitoring
1. Sign in to the Liberator UI as a super-admin.
2. Click **System Monitoring** in the top navigation.
If the menu item doesn't appear, your account doesn't have super-admin privileges. Contact your CloudQuant administrator.
## Tabs
System Monitoring is organized into five tabs. The first four are backed by Prometheus and share a global **time-range selector** (`6h` / `24h`) in the upper-right. The fifth is backed by the entitlements database and uses its own dedicated `1d / 1w / 1m / 1y` selector.
### Cluster
Real-time cluster health from the Liberator gateway, application pods, and host nodes.
What you'll see:
* Gateway request rate and latency percentiles (`p50` / `p95` / `p99`)
* Per-pod CPU and memory utilization for Liberator components
* Per-node CPU, memory, and filesystem utilization
* Data-cache worker pool status
Watch **volume / disk usage** on mounted filesystems. Sustained usage at or above roughly **85%** on a volume warrants immediate attention to avoid query failures from insufficient write space.
Use this tab to answer: *"Is the cluster behaving normally right now, and if not, where is the problem?"*
### Queue
The Liberator **waiting room** queue — how many requests are queued, how long they've been waiting, and which users own them.
What you'll see:
* Active connections (in flight) and queued connections (waiting)
* Per-user breakdown of queue occupancy
* Maximum in-flight query duration (a useful early-warning signal)
The `CQAIOps` service account often appears as a high-volume user; that reflects automated platform monitoring and is expected.
Use this tab to answer: *"Is anyone being blocked, and by whom?"*
### Datasets
Dataset access patterns over the selected time range.
What you'll see:
* Top datasets by query count
* Top datasets by bytes returned
* Distribution of access by client (Python, REST, Excel, etc.)
Use this tab for capacity planning and detecting anomalous access patterns (e.g. a previously dormant dataset suddenly receiving heavy traffic).
### Long queries
The slowest individual queries in the selected window.
What you'll see:
* A ranked list of queries with execution time, user, dataset, and `from` / `to` window
* Click-through to see the full query text and result-set size
Use this tab to find candidates for query rewriting, dataset re-partitioning, or user education.
### Usage
License utilization from the entitlements database. Distinct from the other tabs in two ways:
1. **Different selector.** This tab exposes `1d / 1w / 1m / 1y` windows instead of the Prometheus `6h / 24h`, because license utilization is measured against per-contract caps that operate on much longer windows.
2. **Different backing store.** Numbers come from the entitlements database, not Prometheus, so they survive Prometheus retention rollovers and reflect contract truth.
What you'll see:
* Active vs. licensed seat count, by license tier
* Per-dataset utilization vs. contract caps
* Trend lines that make it easy to spot accounts approaching their limits
* **Most queried datasets** — use this to prioritize [cache pre-generation](/administration/cache-pre-generation)
The `1d` / `1w` selector labels on this tab may still reflect monthly aggregation in the backing entitlements store in some releases. Treat long-window utilization as directional until label semantics match the aggregation period in your environment.
For **per-query** records (who queried which dataset, with which `as_of` / `back_to`, rows returned, and outcome), Super Admins can query the `usage_log` dataset. See [Query usage log](/administration/usage-log).
## Grafana integration
The action in the upper-right of every tab opens the **Grafana Integration** dialog. Super-admins can use it to:
* **Issue Bearer tokens** for external Prometheus consumers (the full token is shown exactly once at creation, so copy it immediately).
* **List existing tokens** with their issue time and issuing user.
* **Revoke tokens** that are no longer needed or may have leaked.
See the full setup walkthrough in the [Grafana integration guide](/integrations/grafana).
## How the data flows
```
┌────────────────────────────────────────────────────────────────────┐
│ Liberator UI │
│ ┌────────────────────────┐ ┌──────────────────────────────┐ │
│ │ Cluster / Queue / │ │ Usage tab │ │
│ │ Datasets / Long Q │ │ │ │
│ └──┬─────────────────────┘ └──┬───────────────────────────┘ │
└─────┼────────────────────────────┼────────────────────────────────┘
│ /metrics-api/* (OIDC cookie)│ /admin-api/entitlements/*
▼ ▼
┌──────────────┐ ┌──────────────────┐
│ Prometheus │ │ Entitlements DB │
│ (read-only) │ │ (PostgreSQL) │
└──────────────┘ └──────────────────┘
▲
│ /metrics-api-bearer/* (Bearer token)
│
┌──────────────┐
│ External │
│ Grafana, │
│ Federation │
└──────────────┘
```
Both Prometheus-fronted endpoints (`/metrics-api/*` for the in-product UI and `/metrics-api-bearer/*` for external consumers) expose the same read-only subset of the Prometheus HTTP API. The OIDC-fronted route is what the in-product tabs use; the Bearer-fronted route is what Grafana and federated Prometheus servers use.
# What's New in Liberator 2.1
Source: https://knowledge.cloudquant.com/whats-new/liberator-2.1
Release notes for the CloudQuant Data Liberator 2.1 release — enterprise SSO, concurrent query limits, broader file and date-format support
# What's new in Liberator 2.1
Liberator 2.1 brings two large workstreams together into a single release: **enterprise SSO** for the Liberator UI, and **concurrent-query governance** so a single user can no longer monopolize platform capacity.
It also picks up a long tail of dataset-quality improvements — broader date-format coverage, a new file type, better handling of records containing special characters, and **[required timestamps on all queries](/python-guide/required-timestamps)**.
## Highlights
The Liberator UI now supports standards-based single sign-on (SAML/OIDC), including federation with your existing identity provider and a customizable login page.
Per-user concurrent-query caps prevent runaway clients from starving capacity.
Liberator now ingests pipe-separated value (`.psv`) files alongside CSV and Parquet, with the same date-format detection improvements applied to all file types.
## Enterprise SSO
Liberator 2.1 promotes a standards-based **single sign-on** flow to the default authentication path for the Liberator UI:
* **Federation with your identity provider** — your users sign into Liberator using their existing corporate SSO session. There's nothing to configure on the user's side, since CloudQuant wires the federation up for your tenant.
* **Customizable login page** — the login page can be styled to match your organization's branding rather than the default look.
* **Logout reliability** — fixed a bug where clicking *Log out* would leave a stale session in some cases, causing the user to be silently signed back in on the next page navigation. The sign-out flow now consistently clears the session on both the gateway and the identity provider.
## Concurrent query limits
A single user issuing dozens of long-running queries can no longer monopolize platform capacity. Liberator now enforces a per-user concurrent-query cap; requests above the cap are queued in the waiting room and processed FIFO. The cap is visible to administrators via the **System Monitoring → Queue** tab introduced in 2.2.
## New file / data-format support
* **PSV (pipe-separated value) files** — Liberator now treats `.psv` as a first-class file type alongside `.csv` and `.parquet`. The auto-detection logic mirrors CSV: header row, type inference, configurable null sentinel.
* **Additional date-format support** — Parquet file uploads with date columns that previously failed during the date-processing step (mixed `YYYY-MM-DD` / `MM/DD/YYYY` / Unix-epoch columns in the same file) now ingest cleanly. The detection is per-column, not per-file.
* **Special-character handling in records** — records containing characters that previously needed manual escaping (multi-line strings, embedded quotes, non-ASCII symbols) round-trip through the API unchanged.
## Performance & reliability
* **Faster small-query response times** — reused connections between Liberator's query and cache tiers shave latency off small queries and reduce pressure on the cache tier under heavy concurrent load.
* **Better diagnostics on cache miss** — when the platform misses cache on a symbol lookup it now records the requested symbol set, making "why is this symbol always being recomputed?" answerable from logs alone.
## Bug fixes
* **Options dataset onboarding** — a previously unsupported options intraday-history dataset is now onboardable through the standard ingestion path.
* **Pivot on uploaded files** — pivot operations on Liberator UI–uploaded files that worked in earlier releases but stopped after a recent cut are working again.
* **Dataset descriptions show display names** — the Liberator UI's dataset description card now renders the dataset's friendly display name instead of falling back to the internal slug.
* **S3-backed dataset date parsing** — fixed a date-parse path that mis-handled certain S3-backed datasets with mixed date-column formats.
## Subsequent fixes shipped in 2.1.1
Some issues that surfaced after 2.1 were fixed in the [2.1.1 patch](/whats-new/liberator-2.1.1), most notably a reliability issue under sustained SQL-source load.
# What's New in Liberator 2.1.1
Source: https://knowledge.cloudquant.com/whats-new/liberator-2.1.1
Patch-release notes for CloudQuant Data Liberator 2.1.1 — reliability under heavy SQL-source load, configurable branding, improved error messages, PCAP FIX delegate
# What's new in Liberator 2.1.1
Liberator 2.1.1 is a **patch release** on top of [2.1](/whats-new/liberator-2.1).
The headline change is a reliability fix that surfaced under sustained SQL data-source load. The remainder of the release is customer-driven polish: per-organization branding controls, friendlier error messages, and a new **PCAP FIX delegate** for low-latency tick consumers.
## Highlights
A blocking call in the data-cache tier could stall the platform under sustained SQL-source ingestion. Fix moves the call off the request-handling path.
The Liberator UI nav-bar logo and login-page logo can now be customized per organization, so your own branding appears in front of your users.
New delegate path for consuming FIX-encoded tick data captured from PCAP, alongside the existing Liberator delegate types.
Liberator error responses now surface root-cause hints rather than a raw exception chain.
## Reliability improvements
* **Stall under sustained SQL-source load** — under sustained ingestion against a SQL-backed datasource, a blocking call in the data-cache tier could occupy the request-handling path and cause the platform to go unresponsive for tens of seconds at a time. The fix moves the blocking call onto a background worker so request handling stays responsive while the SQL backend is busy.
* **Mapping-data fetch resilience** — fixed a regression where mapping-data fetches could error with *"No mapping data returned for ..."* even when the upstream data was present. The fetch retries now distinguish between transient transport errors (which retry) and structurally empty responses (which don't).
* **Force-regen no longer silently returns 0 rows** — the Python client's `force_regen=True` path could return 0 rows when an underlying data-cache regeneration failed silently. Regenerations now surface failures as exceptions rather than returning an empty result.
## Configurable branding
Per-organization branding is now supported across the Liberator UI:
* **Login-page logo**
* **Liberator UI nav-bar logo**
Both default to the CloudQuant logo unless your organization has requested a custom configuration. Contact your CloudQuant account team to set up custom branding for your tenant.
## SSO for the Liberator UI
The enterprise SSO integration introduced in 2.1 is now wired into the **Liberator UI** login flow as well. Federated users see a single "Sign in" button on the login page. Clicking it bypasses the username/password prompt and authenticates via the corporate SSO session if one is already active in the browser.
## PCAP FIX data delegate
Liberator now ships a **PCAP FIX delegate**, a new delegate type that ingests FIX-protocol messages from PCAP captures, decoded and presented to the standard Liberator query surface. This is intended for low-latency tick consumers that capture exchange feeds at the wire level and want to query them through the same API as their historical datasets.
## Improved error messages
Liberator's error responses across the Python client, REST API, and Liberator UI have been re-paved so that:
* The *root* cause is reported first, with the call-site trace as supplementary context (previously the call-site trace was first, burying the actual cause).
* Common configuration mistakes (missing entitlement, malformed date parameter, unknown dataset) get plain-English explanations instead of a raw exception class name.
* Network-transport errors are distinguishable from server-side errors in the response shape, so client libraries can apply different retry behavior to each.
# What's New in Liberator 2.2
Source: https://knowledge.cloudquant.com/whats-new/liberator-2.2
Release notes for the CloudQuant Data Liberator 2.2 release — System Monitoring, Grafana integration, scheduled cache creation, ETL on-the-fly mapping UI, and reliability improvements
# What's new in Liberator 2.2
Liberator 2.2 focuses on **observability and operability**. The release adds an in-product System Monitoring page for super-admins and a standardized way to plug external Grafana instances into the underlying metrics. It also introduces **scheduled cache creation**, which removes cold-start latency for predictable access patterns, and the foundations of an **ETL on-the-fly dataset mapping** workflow.
It also lands a reliability fix that prevents long-running query workers from getting stranded across platform restarts.
## Highlights
A new admin-only page surfacing cluster health, queue depth, dataset access patterns, long-running queries, and license utilization.
A Bearer-authenticated, read-only metrics endpoint at `/metrics-api-bearer/`, with token issuance and revocation built into the Liberator UI.
Pre-generate dataset caches on a schedule so users never hit a cold-cache penalty for predictable access patterns.
Define cross-dataset value correspondences and enrich queries with `translate=true`.
Query workers no longer get stranded on stale connections when the platform restarts behind them.
System Monitoring's Usage tab uses its own license-appropriate time window, and tab state is preserved as users navigate.
## System monitoring
A new super-admin-only **System Monitoring** page consolidates the operational signals previously scattered across logs and ad-hoc queries. Five tabs:
| Tab | Time-range selector | What it answers |
| ---------------- | ------------------- | -------------------------------------------------------------- |
| **Cluster** | `6h / 24h` | Is the cluster behaving normally right now? |
| **Queue** | `6h / 24h` | Is anyone being blocked, and by whom? |
| **Datasets** | `6h / 24h` | What are users actually querying? |
| **Long Queries** | `6h / 24h` | Which individual queries are slow enough to warrant attention? |
| **Usage** | `1d / 1w / 1m / 1y` | How close are we to our license caps? |
See the [System Monitoring overview](/system-monitoring/overview) for a walkthrough.
## Grafana integration
CloudQuant Data Liberator now exposes a standardized, Bearer-authenticated metrics endpoint at `https:///metrics-api-bearer/`. The endpoint speaks the full standard Prometheus HTTP API surface (`/api/v1/query`, `/api/v1/query_range`, `/api/v1/series`, `/api/v1/labels`, `/api/v1/label//values`, `/api/v1/status/buildinfo`, `/federate`). Any off-the-shelf Prometheus client — Grafana, Mimir, federated Prometheus, custom scripts — can consume it without modification.
Token lifecycle is fully self-service from the Liberator UI:
1. **Issue** — a super-admin clicks **Generate token** in the **Grafana Integration** dialog. The full token value is shown exactly once.
2. **Use** — point Grafana at `/metrics-api-bearer` with `Authorization: Bearer cqm_` as a custom HTTP header.
3. **Rotate** — issue a new token, switch Grafana to it, verify, then revoke the old one (zero-downtime).
4. **Revoke** — same dialog, list view.
See the full walkthrough at [Grafana integration](/integrations/grafana).
## Scheduled cache creation
For datasets with predictable access patterns (daily morning reports, weekly market-open snapshots), Liberator now supports **scheduled cache creation**. Super-admins set a cron-style schedule per dataset directly in the Liberator UI's caching settings. The platform pre-generates the cache slice before users query it, so the first read of the day hits a warm cache rather than triggering a generation.
See the [cache pre-generation guide](/administration/cache-pre-generation) for the full Super Admin workflow.
## ETL mapping UI
Liberator 2.2 lays the foundation for the **ETL on-the-fly** workflow. Analysts define cross-dataset value correspondences (e.g. "the `ticker` column in Dataset A maps to the `symbol` column in Dataset B, with these reconciliations"). Liberator then surfaces a *virtual unified column* across both datasets at query time, without a heavyweight pre-materialization step.
What lands in 2.2:
* **ETL mapping UI design** — UX research and component specs.
* **Data Translation Mapping** — the underlying admin-API surface for storing and retrieving mapping definitions.
* **Dataset Field Mapping UI** — the in-Liberator-UI workflow for defining correspondences and previewing the virtual unified column. See [Dataset field mapping](/administration/dataset-field-mapping).
Full GA of the mapping workflow continues in later releases; see [What's New in Liberator 2.3](/whats-new/liberator-2.3) for related AI and connectivity work shipped after 2.2.
## Reliability improvements
A user-facing reliability bug is fixed in 2.2: previously, after a behind-the-scenes platform restart, a small number of long-running queries could hang indefinitely rather than reconnecting. In 2.2 those workers detect the broken connection within a short keepalive window and reconnect automatically, so long-running queries continue cleanly across platform restarts instead of stalling.
## UI changes
* **Usage tab time-range selector.** The global `6h / 24h` selector is now hidden when the **Usage** tab is active. Usage has its own `1d / 1w / 1m / 1y` window because license utilization is measured against per-contract caps that don't make sense on a 6-hour window.
* **Controlled Tabs.** The System Monitoring page's tab component is now fully controlled, so per-tab state survives switches without remounting, and the token-rotation flow triggers a clean cache reset across all tabs.
## Also in 2.2
Smaller items in the same release:
* **OneDrive URL share-link paste** — paste a full OneDrive or SharePoint share URL into a connection form; see [SharePoint and OneDrive](/datasource-config/sharepoint-onedrive).
* **Email verification** — new accounts receive an email verification message when provisioned.
* **Connection-form example text** — UX hints in connection-source forms so users know what shape of input each field expects (host, bucket name, region, etc.) before they make a mistake.
## Compatibility
There are no breaking changes to the client libraries (Python, REST, JavaScript, Excel, etc.) or to the dataset query surface, so existing code and queries continue to work unchanged. The new features in 2.2 are surfaced through the Liberator UI and the Bearer-token metrics endpoint.
To start using the Grafana integration, a super-admin generates a Bearer token from the **Grafana Integration** tab in the Liberator UI. See the [Grafana integration](/integrations/grafana) guide for the full walkthrough.
# What's New in Liberator 2.3
Source: https://knowledge.cloudquant.com/whats-new/liberator-2.3
Release notes for the CloudQuant Data Liberator 2.3 release — AI-driven dataset onboarding, AI-driven reporting, HubSpot connectivity, invite-via-email onboarding, SharePoint wildcards, and reliability improvements
# What's new in Liberator 2.3
Liberator 2.3 centers on **AI-driven dataset onboarding and reporting**, **expanded connectivity**, and **smoother account onboarding**. Headline capabilities include an AI Assisted Create Dataset flow that fills connection and column metadata from a plain-language description, an Admin UI AI Reporting panel for usage and entitlement reports, and AI-generated column descriptions. The release also adds a HubSpot connector for CRM analytics, wildcard support for OneDrive and SharePoint file feeds, and invite-via-email user provisioning with a forced password change on first login. A round of reliability fixes covers date parsing, query workers, and packet-capture ingest.
## Highlights
Describe a source in plain language; Liberator tests the connection and fills the Create Dataset wizard for you to review.
Ask plain-English questions about usage, users, and entitlements in the Admin UI, with quick-action reports and Excel exports.
Onboarding and metadata edits can suggest human-readable column descriptions, ranked by how often each description is used in your environment.
Connect HubSpot CRM objects into Liberator and query them through the same Liberator surface as market and file-backed datasets.
Create users with an email invite, send password-reset mail from the Users page, and require a new password on first login.
Pattern-match multi-file and dated folder feeds without enumerating every path.
Connect ChatGPT, Copilot Studio, and Claude Desktop to Liberator over MCP.
## AI-driven reporting
Liberator 2.3 adds an **AI Reporting** panel (BETA) to the Admin UI. Admins can ask about platform usage, users, permissions, and entitlements in plain English, or request a downloadable report, instead of building manual queries or filters.
What lands in 2.3:
* **Quick-action reports** — one-click presets for yesterday's usage, today's user report, current entitlements, and month-to-date / last-month usage.
* **Free-form natural language** — ask for a user's usage over a date range; the assistant resolves datasets and returns per-dataset totals (queries, records, bytes, GB).
* **Excel exports** — downloadable `.xlsx` reports from both quick actions and free-form questions.
* **Clarifying prompts** — if a username does not match, the assistant suggests likely accounts instead of failing silently.
See [AI-based reporting](/administration/ai-based-reporting) for the admin workflow.
## AI-driven dataset onboarding
Liberator 2.3 adds an **AI Assisted** path (BETA) to the Create Dataset wizard for every connector type. After you select a source, choose AI Assisted, describe the source (or paste connection details), and Liberator tests the connection and fills the wizard steps for review.
What lands in 2.3:
* **AI Assisted vs Manual** — pick assisted setup or step through fields yourself from the source picker.
* **Natural-language or pasted details** — describe the source or paste credentials, paths, and table names into the chat panel.
* **Auto-filled wizard steps** — connection fields, timezone, date/time detection, sample rows, suggested key, and per-column types, display names, and AI-generated descriptions.
* **Editable before create** — every suggested field can be changed before you create the connection or dataset.
* **Wildcard patterns preserved** — file prefixes such as `folder/prefix_*.csv` carry through unchanged.
* **In-flow help** — ask about wizard concepts without leaving the current onboarding task; off-topic requests are declined.
See [AI-driven dataset onboarding](/administration/ai-dataset-onboarding) for the full workflow.
## AI-generated column descriptions
When you onboard a dataset or edit column metadata, Liberator can **suggest descriptions** grounded in descriptions already used on similar columns in your environment.
What lands in 2.3:
* **AI-assisted suggestions** during the Columns step of dataset onboarding and edit flows.
* **Frequency-based ranking** so the most commonly used descriptions appear first.
* **Tighter relevance filtering** so broad substring matches no longer crowd out useful suggestions.
See [Column description suggestions](/administration/column-description-suggestions) for the admin workflow.
## HubSpot connector
Liberator 2.3 adds a first-class **HubSpot** connection type for CRM objects. Once connected, HubSpot-backed datasets use the same query API and entitlements model as file- and database-backed sources, so analysts can join commercial and market data without a separate export pipeline.
See [HubSpot](/datasource-config/hubspot) for connection setup.
## Invite-via-email and first-login password change
User provisioning no longer requires handing temporary passwords around out of band:
1. **Invite via email** — when creating a user, Super Admins can send an invite email so the recipient completes account setup themselves.
2. **Send reset email** — from the Users page, trigger a password-reset email for an existing account.
3. **Forced password change on first login** — newly provisioned users must set their own password before using Liberator.
See [Managing users and entitlements](/administration/managing-users-and-entitlements#invite-via-email).
## SharePoint and OneDrive wildcards
Building on the share-link paste flow introduced in 2.2, Liberator 2.3 adds **wildcard matching** for OneDrive and SharePoint sources. Recurring or partitioned file feeds (for example, `reports/2026-*/trades_*.csv`) can be onboarded without listing every folder or file.
See [SharePoint and OneDrive](/datasource-config/sharepoint-onedrive).
## MCP clients
Connect external AI assistants to Liberator over MCP:
* **ChatGPT** — connect ChatGPT to Liberator with OAuth. See [Connecting Liberator to ChatGPT](/integrations/chatgpt).
* **Microsoft Copilot Studio** — add Liberator as an MCP tool with OAuth 2.0 Dynamic discovery. See [Microsoft Copilot Studio](/integrations/copilot).
* **Claude Desktop** — install the Liberator `.mcpb` extension bundle. See [Claude Desktop](/integrations/claude).
All three MCP client guides live under the **Integrations** tab.
## Connection and admin polish
Smaller usability improvements in the same release:
* **`liberator.metadata()`** — Python SDK helper for `POST /liberator/metadata`, returning concordance-backed row/symbol counts and date spans. See [Dataset metadata](/api-reference/concepts/dataset-metadata).
* **Disable data sources** — administrators can hide deprecated or unused source types from the connection selection list.
* **Original source file name** — the Dataset & Connection modal shows the original file name for better traceability.
* **Whitespace trimming** — leading and trailing spaces are stripped from connection and dataset form fields automatically.
* **Sort resets to page one** — sorting the Connections & Datasets list returns you to the first page of results.
* **System Monitoring charts** — legends stay contained and hover tooltips sort by value for easier reading.
## Reliability improvements
* **Date/time format handling** — fixes for format selection UX and regressions around forward-slash date fields (including Oracle sources).
* **Query worker recovery** — workers that die abruptly no longer leave clients hanging indefinitely on incomplete responses.
* **PCAP Linux SLL2** — packet captures with Linux cooked-mode v2 link-layer headers (`link_type=276`) decode correctly for FIX tick ingest.
* **Fine-grained entitlements enforcement** — end-to-end entitlement checks tightened so UI-visible access matches what the query path allows.
## Compatibility
There are no breaking changes to the client libraries (Python, REST, JavaScript, Excel, etc.) or to the dataset query surface. Existing code and queries continue to work unchanged. New capabilities in 2.3 are additive through the Liberator UI, connection types, and AI assistant surfaces.
# What's New in Liberator 2.4
Source: https://knowledge.cloudquant.com/whats-new/liberator-2.4
Release notes for the CloudQuant Data Liberator 2.4 release — symbol pattern filters, usage_log, clearer entitlement errors, dataset health cards, Excel plugin in the UI, and more reliable queries
# What's new in Liberator 2.4
Liberator 2.4 centers on **more precise queries**, **clearer dataset health**, and **safer limits**. Headline capabilities include wildcard / pattern filters on the symbol (key) parameter, a Super-Admin-only `usage_log` dataset of per-query records, and descriptive entitlement-limit errors. The release also adds a per-dataset cap on how many days a single query can span, Concordance-backed dataset catalog cards, the CloudQuant Excel plugin in the Liberator UI, and a round of connection and time-range reliability fixes.
## Highlights
Use `%` wildcards on `symbols` / `keys` to pull a product family or naming pattern without listing every contract.
Limit violations say which cap was hit and what to change, instead of a generic "not entitled."
Admins can set a maximum number of days a single query may span on a dataset, so one wide request cannot stall everyone else.
Catalog cards show date range, symbol count, status, and last updated — in the UI and via `liberator.datasets(details=True)`.
Download and access the CloudQuant Excel plugin from the Liberator client, without a separately shared installer.
Create and enroll a user in one flow. Invite emails include the username to sign in with.
Super Admins query `usage_log` like any other dataset — who queried what, which window, rows returned, and outcome.
## Symbol and key pattern filters
`symbols` (and the `keys` synonym) accept SQL `LIKE` patterns. An unescaped `%` is a wildcard for any sequence of characters:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name='energy_contracts',
symbols='PWR%', # every key that starts with PWR
back_to='2026-08-01',
as_of='2026-08-02',
))
```
You can mix exact keys and patterns in one list. To match a literal percent character in a key, escape it as `\%`.
See [Symbols and key fields](/python-guide/symbols-and-key-fields#pattern-filters-24) and the [query parameters reference](/api-reference/concepts/query-parameters#dataset--symbol-parameters).
## Clearer entitlement errors and query day-range limits
When a query is rejected because of an entitlement or usage limit, Liberator now returns a **specific message**: which limit applied, the configured value, and what to change. A row-limit miss no longer looks the same as a missing entitlement.
Super Admins can also set a **maximum number of days** a single query may span on a given dataset (or dataset group). Use this on high-volume sources so a misconfigured `back_to` / `as_of` window cannot pin query workers. If a query exceeds the cap, the caller gets the same class of descriptive error.
See [Fine-grained permissions](/administration/fine-grained-permissions).
## Dataset health cards and symbol discovery
The dataset catalog now surfaces Concordance-backed health on each dataset:
* Date range covered
* Symbol count
* Status (for example, Ready)
* Last updated
The same fields are included when you call `liberator.datasets(schema=True, details=True)`. Administrators can configure how often symbol discovery refreshes, so newly added keys appear on cards without waiting for a fixed backend job.
See [Listing datasets](/api-reference/concepts/listing-datasets) and [Checking dataset access](/python-guide/dataset-access).
## Excel plugin in the Liberator UI
The CloudQuant Excel plugin is available from the Liberator client experience. Users can find and download it in the UI instead of waiting on a separately shared installer. Account setup (User ID, token, Base URL) is unchanged.
See [Configuring your account](/api-reference/excel-plugin/configuring-account).
## User onboarding
* **Combined create-and-enroll** — Super Admins create a user and add them to Liberator permissions in one step, instead of Create User followed by Add Users.
* **Username in invite and verification email** — new-account mail includes the username the recipient should sign in with.
* **Search by company** — the Users list finds people by their company field.
See [Managing users and entitlements](/administration/managing-users-and-entitlements).
## Per-query usage log
Super Admins can query **`usage_log`**, a private Liberator dataset of one row per query per dataset. Each row records the user, the dataset they asked for, the `as_of` / `back_to` they passed, rows returned, and an outcome of `success`, `limit_exceeded`, `not_entitled`, or `error`.
Query it with the standard client (`liberator.query(name='usage_log', ...)`). The log's own `back_to` / `as_of` select **when the usage happened**; the `as_of` / `back_to` **columns** are the window the original caller requested. The symbol / key is the dataset that was queried. Records are retained for at least 30 days.
See [Query usage log](/administration/usage-log).
## Connection and query reliability
* **S3 request style** — during Test Connection, Liberator detects whether the endpoint needs path-style or virtual-hosted addressing (including MinIO, Ceph, and other S3-compatible stores), so you no longer have to set `request_style` by provider. See [Amazon S3](/datasource-config/s3).
* **Database port is required** on Test Connection for PostgreSQL, MySQL, SQL Server, and Oracle.
* **Passwords with special characters** (`%`, `?`, `#`, `@`, `/`, `:`, `&`) connect successfully on database sources.
* **Fixed / manual timestamps** — datasets onboarded with a fixed date-time (no date column) query the correct partition after discovery.
* **Time-range queries** resolve partitions more reliably across the requested window.
* **Future `as_of` on live-append datasets** no longer holds query workers in an empty-fetch loop. Prefer `as_of` at or before "now" unless you intentionally query forward-dated records. See [Usage of the as\_of parameter](/python-guide/as-of-parameter).
* **HubSpot rate limits** — HubSpot-backed queries wait and retry when the CRM API returns a rate-limit error. See [HubSpot](/datasource-config/hubspot).
* **Date-format ranking** — onboarding suggestions prefer ordinary calendar formats over day-of-year when the column contains month/day/year values.
## Compatibility
There are no breaking changes to the client libraries (Python, REST, JavaScript, Excel, etc.) or to the dataset query surface. Existing exact-symbol queries continue to work. Pattern filters, day-range caps, catalog-card fields, and `usage_log` are additive.
# What's New
Source: https://knowledge.cloudquant.com/whats-new/overview
Release notes and change history for the CloudQuant Data Liberator platform
# What's new
Release notes for the CloudQuant Data Liberator platform. Each version page summarizes the user-visible changes in that release.
**Latest.** Symbol pattern filters, per-query `usage_log`, clearer entitlement errors, query day-range caps, dataset health cards, Excel plugin in the UI, and more reliable connections.
AI-driven dataset onboarding, AI-driven reporting, HubSpot connectivity, invite-via-email onboarding, SharePoint/OneDrive wildcards, and reliability improvements.
In-product System Monitoring, Grafana integration, scheduled cache creation, ETL on-the-fly mapping UI, and reliability improvements.
Stability patch. Reliability improvements under heavy SQL-source load, configurable branding, friendlier error messages, and a new PCAP FIX delegate.
Major release. Enterprise SSO, concurrent query limits, PSV file support, and broader date-format coverage.
## Versioning
Liberator uses **semantic-style versioning** (`MAJOR.MINOR.PATCH`):
* **MAJOR / MINOR** versions (e.g. `2.3` → `2.4`) bundle new features alongside fixes.
* **PATCH** versions (e.g. `2.1` → `2.1.1`) are restricted to fixes and small enhancements.
CloudQuant manages the rollout of new versions to your environment. Reach out to your CloudQuant account team if you have questions about when a specific change reaches you.
# AI-Based Reporting
Source: https://knowledge.cloudquant.com/administration/ai-based-reporting
Ask natural-language questions about Liberator usage, users, and entitlements from the Admin UI
# AI-based reporting
The **AI Reporting** panel is a natural-language assistant in the Liberator Admin UI. Admins can ask about platform usage, users, permissions, and entitlements in plain English, or request a downloadable report, without building manual queries or filters.
Available in Liberator **2.3** and later. The panel is labeled **BETA**.
## Who can use it
AI-based reporting is an **Admin / Super Admin** capability in the Liberator Admin UI. Results respect the same platform data the admin portal already uses for usage and entitlement views.
## Quick-action reports
The panel includes preset buttons that run canned reports and return a narrative summary (and an export when applicable):
| Quick action | What it returns |
| ----------------------------- | ----------------------------------------------- |
| **Usage summary (yesterday)** | High-level usage narrative for the previous day |
| **Usage report (yesterday)** | Detailed usage report for the previous day |
| **User report (today)** | User activity report for the current day |
| **Current entitlements** | Snapshot of current entitlement grants |
| **Usage report (last month)** | Usage for the previous calendar month |
| **Usage report (this month)** | Usage for the current calendar month |
## Free-form questions
Type requests in natural language, for example:
* "Give me the usage for `` for the second week of February."
* "Which users were most active yesterday by query count?"
* "Show entitlements for ``."
The assistant resolves the request against live data: it identifies the relevant dataset(s), computes per-dataset or per-user totals (queries, records, bytes, GB), and can generate a downloadable Excel (`.xlsx`) export.
### Username clarification
If a typed username does not match a real account, the assistant asks you to confirm or correct it and suggests likely matches. After you confirm, it continues with the corrected user without requiring you to retype the full request.
### Empty periods
If no usage occurred in the requested window, the assistant still produces a valid (header-only) export and explains that there was no activity, rather than returning an error.
## Excel exports
Reports from quick actions or free-form questions can be downloaded as `.xlsx` files. The chat response includes the filename and file size. Typical columns include user, dataset, query count, record count, bytes, and gigabytes, matching the summary shown in the panel.
## Related
Per-query records (2.4+) — who queried which dataset, window, rows, and outcome
Provision users and grant dataset access
Liberator 2.3 release notes
# AI-Driven Dataset Onboarding
Source: https://knowledge.cloudquant.com/administration/ai-dataset-onboarding
Use the AI Assisted flow in the Create Dataset wizard to describe a source in plain language and auto-fill connection and dataset settings
# AI-driven dataset onboarding
AI Onboarding is an assisted setup flow built into Liberator's **Create Dataset** wizard. After you pick a source type, choose **AI Assisted** to describe your data source in plain language (or paste connection details). The assistant tests the connection and fills the standard wizard steps for you to review before creating.
Available in Liberator **2.3** and later. The flow is labeled **BETA**. Manual setup remains available for every connector.
## Who can use it
Admins and Super Admins who can create connections and datasets. AI Assisted is offered for every connector type, including S3, Snowflake, SFTP, SQL Server, MySQL, Oracle, PostgreSQL, OneDrive, SharePoint, Azure Blob, Google Cloud Storage, CIFS, FTPS, Local File, and HubSpot.
## Choose AI Assisted or Manual
1. Go to **Datasets → Add Dataset** (or start from **Connections**).
2. Select a source type.
3. Choose a setup flow:
| Flow | When to use it |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| **AI Assisted (BETA)** | Paste credentials and a short description; let Liberator fill connection, config, dataset details, and column metadata |
| **Manual** | Step through every connection and dataset field yourself |
## Describe your source
Choosing **AI Assisted** opens a chat panel with a prompt such as **Describe your \ Dataset**. You can:
* Paste raw connection details (host, keys, bucket, folder/prefix, table name, and so on)
* Describe the source in plain language
* Include wildcard or prefix file patterns (for example, `reports/mag_7_fake_*.csv`) — patterns are preserved exactly in the folder/prefix field
If you do not include a dataset name, the assistant asks for one before continuing. The dataset name becomes the queryable identifier.
## What the assistant fills in
On a successful connection test, the assistant populates the standard wizard steps:
| Wizard step | What AI fills |
| ------------------- | ------------------------------------------------------------------------------ |
| **Basic Info** | Dataset name (if provided or confirmed in chat) |
| **Connection** | Credentials, endpoint, bucket/account, and related connection fields |
| **Configuration** | Default timezone, detected date/time column(s) and format, sample-rows preview |
| **Dataset Details** | Suggested queryable key |
| **Column Metadata** | Column type, display name, and an AI-generated description for each column |
| **Webhooks** | Left for you to configure if needed |
Every AI-filled field stays editable. Nothing is locked until you click **Create Connection** / **Create Dataset** (or the equivalent continue action) at each step.
## Review before creating
The assistant asks you to review and edit anything that looks off. Typical checks:
* Connection test reported success
* Folder/prefix or table matches the intended source
* Date/time column and format look correct
* Sample rows show the expected data
* Column descriptions and the queryable key are usable as-is or edited
## In-flow help
You can ask clarifying questions about the current wizard (for example, what a translation column is) without restarting onboarding. The assistant stays scoped to connecting and configuring **this** dataset — off-topic or destructive requests (such as deleting other datasets) are declined and the conversation is redirected to the outstanding connection details.
## After creation
The new dataset appears in the **Datasets** list with the name you confirmed and the correct source-type badge. You can still edit connection or dataset settings later through the normal Liberator UI.
## Related
AI-assisted and frequency-ranked column descriptions
Connection + dataset architecture and source types
Natural-language usage and entitlement reports
Liberator 2.3 release notes
# Cache Pre-Generation
Source: https://knowledge.cloudquant.com/administration/cache-pre-generation
Schedule dataset cache creation ahead of time so users query warm Parquet caches instead of cold source systems
# Cache pre-generation
By default, Liberator builds its cache the first time a query runs (cache warming). For large datasets or slow source databases, that first query can take a long time. **Pre-generated cache** lets Super Admins schedule cache creation ahead of time so data is ready when users query it.
Cache files are written in **Parquet** format (replacing the older Arrow cache format) and can be stored in **Amazon S3** or **Google Cloud Storage**. Cached data does not need to live on the Liberator server. Parquet is typically much more compact than the previous Arrow caches, which reduces storage cost at scale.
Super Admin role is required to configure cache storage connections and pre-generation settings.
## Step 1 — Create a cache storage connection
Before enabling pre-generated cache on a dataset, create a cache storage destination.
Go to **Connections → Add Connection**.
Select **Cache Storage** as the connection type.
Select your storage provider (**S3** is supported today; confirm GCS availability with your CloudQuant account team if needed). Enter the bucket name and credentials.
Click **Test Connection**, then **Save**.
## Step 2 — Enable pre-generated cache on a dataset
Open the dataset and click **Edit**.
Scroll to **Advanced Options** and toggle on **Pre-Generated Cache**.
Select **Parquet** as the cache format (recommended). Under **Destination**, select the cache storage connection from Step 1.
Click **Test** next to the destination to confirm connectivity, then save.
## Step 3 — Configure the cache window
| Option | Use when |
| -------------------- | ---------------------------------------------------------------------------------------- |
| **Rolling window** | Users mostly query recent data; keeps the last N days cached and advances automatically. |
| **Fixed date range** | You need a known historical slice that does not change. |
| **Full dataset** | You want complete coverage and the dataset is bounded in size. |
Enter the number of days (rolling) or the start/end dates (fixed), then save.
Long or open-ended retention periods trigger a **storage impact warning** in the UI. Review projected volume before saving — large windows in Parquet can accumulate significant object storage over time.
## Step 4 — Run or schedule generation
**Scheduled:** Once a schedule is configured on the dataset, cache generation runs automatically. No further action is required.
**Manual:** Open the dataset and click **Trigger Cache Pre-Generation**. A status indicator shows whether the job is in progress or complete.
## What users experience
After pre-generated cache is populated, queries against that dataset are much faster because Liberator reads the warm cache instead of the source system. This matters most for large SQL-backed datasets where the first cold query previously took minutes.
## Tips
* Use **rolling window** for datasets queried over recent windows (for example, the last 30 days of market data).
* Use **fixed date range** for historical snapshots that do not change.
* Use **full dataset** only when storage budget and dataset size are understood.
* The **Most queried datasets** view on [System Monitoring](/system-monitoring/overview) helps prioritize which datasets to pre-generate.
# Column Description Suggestions
Source: https://knowledge.cloudquant.com/administration/column-description-suggestions
Reuse and AI-assisted column descriptions when onboarding or editing dataset metadata
# Column description suggestions
When you onboard a dataset or edit column metadata, Liberator can **suggest descriptions** already in use on similar columns in your environment, ranked by how often each description appears. Accept a suggestion as-is, edit it, or write your own.
In Liberator **2.3** and later, suggestions can also be **AI-assisted**: Liberator proposes human-readable descriptions for cryptic or undocumented columns, then ranks them alongside your existing description library.
During **AI Assisted** dataset creation, column descriptions are generated as part of the wizard fill. See [AI-driven dataset onboarding](/administration/ai-dataset-onboarding).
This keeps documentation consistent across related datasets and speeds up onboarding for common fields (timestamps, tickers, prices, identifiers).
Admin privileges or higher are required. Suggestions improve as more datasets with descriptions exist in your environment.
## Use suggested descriptions
Go to **Datasets → Add Dataset** or **Edit** on an existing dataset.
Proceed to the **Columns** step.
For each column, Liberator shows likely description matches ordered by frequency of use.
Click a suggestion to apply it, edit the text in place, or type a new description manually.
Complete the wizard and save. New and edited descriptions feed future suggestions.
## Tips
* Use **consistent terminology** (for example, always "Trade execution timestamp (UTC)") so rankings stay useful.
* **Edit** close matches instead of creating near-duplicates that compete in frequency ranking.
* Treat low-frequency suggestions cautiously — they may not generalize.
## FAQ
**No suggestion for a column?**\
The field is blank until you enter one manually. It will be available for future datasets with similarly named columns.
**Can I ignore suggestions?**\
Yes. Suggestions are advisory only.
**Do suggestions cross public and private datasets?**\
Yes. Descriptions are metadata drawn from all datasets in your environment, not from underlying row data.
# Dataset Field Mapping
Source: https://knowledge.cloudquant.com/administration/dataset-field-mapping
Map source dataset columns to reference lookup tables and enrich query results with translate=true
# Dataset field mapping
Dataset field mapping defines a relationship between a **source** dataset and one or more **reference** (lookup) datasets. When a user queries the source with translation enabled, Liberator enriches values from the reference data at query time. Source data is never modified.
**Example:** A transactions dataset stores country codes (`US`, `DE`, `JP`). A reference dataset maps codes to English country names. With mapping configured, `translate=true` returns the country name automatically.
You can attach **multiple lookup tables** to one source dataset (products, locations, employees, and so on) and resolve them in a single query.
Admin privileges or higher are required. The reference dataset must already exist in Liberator before you map it.
## Step 1 — Prepare the reference dataset
Onboard lookup data as a standard dataset (CSV, database table, etc.). Include at least:
* A **key** column to match on (for example, `country_code`)
* One or more **output** columns (for example, `english_name`)
## Step 2 — Add translation columns
Navigate to the source dataset users will query and click **Edit**.
Go to the **Columns** step and click **Add Translation Columns**.
## Step 3 — Configure the mapping
In the **Add Translation Columns** dialog:
1. **Select reference dataset** — choose the lookup dataset.
2. **Source column (join key)** — column in the source dataset to match (for example, `country_code`).
3. **Reference column (join key)** — corresponding column in the reference dataset (for example, `alpha_2_code`).
4. **Output column(s)** — reference columns to surface in results (for example, `english_name`).
Click **Add**. Repeat for additional lookup tables.
## Step 4 — Save and query
Save the dataset. When querying, pass the `translate` parameter:
| Parameter | Result |
| ----------------- | --------------------------------------------------------- |
| `translate=true` | Returns enriched/translated values from attached lookups. |
| `translate=false` | Returns raw source data only. |
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name="transactions",
symbols=["AAPL"],
back_to="2026-01-01",
as_of="2026-01-31",
translate=True,
))
```
See [Query parameters](/api-reference/concepts/query-parameters) for `translate` in other clients.
## Conflict and missing-value behavior
| Situation | Behavior |
| --------------------------------- | --------------------------------------------------- |
| No match in the lookup table | Output column is `null`; the row is still returned. |
| Multiple entries for the same key | The entry with the **most recent timestamp** wins. |
| Multiple lookup tables | All lookups resolve in one query pass. |
## Maintenance
* Edit mappings from **Edit → Columns** on the source dataset.
* Changes take effect on the next query.
* Deleting a mapping entry causes affected output columns to return `null`.
## Tips
* Keep reference datasets small and focused on mapping data only.
* Split large flat mapping tables into per-topic lookup datasets when possible.
* Test with `translate=true` on a narrow date range before rolling out to users.
# Dataset Visibility
Source: https://knowledge.cloudquant.com/administration/dataset-visibility
Control whether datasets are public (discoverable by all authenticated users) or private (entitlement required)
# Dataset visibility
Every dataset has a **Publicly Available** toggle that controls whether authenticated users can discover and query it without an individual entitlement.
| Visibility | UI toggle | Who can see and query |
| ----------- | -------------------------- | --------------------------------------------------------- |
| **Public** | **Publicly Available** ON | Any authenticated user — no per-user entitlement required |
| **Private** | **Publicly Available** OFF | Only users with an explicit entitlement |
Authentication is always required. Unauthenticated requests never return datasets.
Visibility and entitlements are separate. A **private** dataset still requires an entitlement assignment on the [Users](/administration/managing-users-and-entitlements) page. A **public** dataset is queryable by any signed-in user without that step.
## Who can change visibility
| Role | Can set public / private |
| --------------- | ------------------------ |
| **Super Admin** | Yes |
| **Admin** | Yes |
| **User** | No |
## Set visibility during onboarding
As Admin or Super Admin, go to **Datasets → Add Dataset**.
Proceed through source, connection, and columns.
Near the top of the configuration page, set **Publicly Available** ON (public) or OFF (private).
Complete the wizard and save. The setting takes effect immediately.
## Change visibility on an existing dataset
1. Go to **Datasets** in the admin portal.
2. Use the **Public** column in the list to audit current visibility at a glance.
3. Open the dataset, toggle **Publicly Available**, and save.
Making a public dataset private hides it from non-entitled users immediately.
## API and catalog behavior
* **Private** datasets do not appear in catalog browse or search for users without entitlement.
* **Public** datasets appear in the catalog for any authenticated user.
* Use `liberator.datasets(entitled=True)` to list datasets you can access (entitlements plus public datasets you are allowed to see). See [Checking dataset access](/python-guide/dataset-access) and [Listing datasets](/api-reference/concepts/listing-datasets).
## Pre-generated cache
Visibility does not change cache behavior. Pre-generated cache continues to serve entitled users if you switch a dataset from public to private.
## Recommended practices
* **Public** — shared reference data (calendars, instrument masters) that all users need.
* **Private** — client-specific, team-scoped, or sensitive datasets.
* Audit the **Public** column periodically; test datasets left public during onboarding are a common oversight.
## Troubleshooting
**User cannot see an expected dataset** — If private, confirm entitlement on the Users page. If public, confirm the user can authenticate.
**Dataset visible to users who should not see it** — Set **Publicly Available** OFF and save.
**Toggle not visible** — Confirm you are signed in as Admin or Super Admin.
## Platform datasets
Some datasets are private by design and are not managed with **Publicly Available**. In Liberator **2.4** and later, **`usage_log`** is Super Admin only and is hidden from the standard user catalog. See [Query usage log](/administration/usage-log).
# File Upload Management
Source: https://knowledge.cloudquant.com/administration/file-upload-management
Upload, replace, download, and delete files behind file-upload connections without backend access
# File upload management
The **File Management** panel on a file-upload connection lets Admins manage the physical file behind a dataset: upload, replace, download, or delete — without access to underlying storage.
Supported formats for upload connections include `.csv`, `.tsv`, `.psv`, `.txt`, and `.xlsx`. See [Supported Data Formats](/datasource-config/supported-formats) for the full list.
Admin privileges or higher on the connection are required. The panel appears only on **File Upload** connections, not S3, SharePoint, database, or API sources.
## Open File Management
1. Sign in to the admin portal.
2. Go to **Connections** and open the connection.
3. Scroll to **File Management**.
| Panel state | Meaning | Action |
| ---------------------- | ----------------------- | ------------------------------------- |
| Empty (no file listed) | No file associated yet | **Upload** |
| File name shown | File already associated | **Replace**, **Download**, **Delete** |
## Upload a new file
Use when the connection has no file yet, or after a delete.
The panel shows no file name and an **Upload** control.
Click **Upload**, select the file, and wait for the progress indicator to complete.
Confirm the file name appears and run a small test query on the linked dataset.
If the dataset was newly created, finish column, date, and key configuration before notifying users.
## Replace an existing file
Use when data was refreshed or corrected but dataset name, columns, entitlements, and mappings should stay the same.
The replacement file must use the **same format** and **same column structure** as the original. Added, removed, or renamed columns require re-onboarding instead of replace.
Use **Download** to inspect the current file first.
Click **Replace**, select the new file, and wait for completion.
Query a value you know exists only in the new file to confirm the swap.
If **pre-generated cache** is enabled, trigger **Trigger Cache Pre-Generation** or wait for the next scheduled run so queries do not serve stale cache from the old file.
## After upload or replace
| Aspect | Behavior |
| ----------------------- | ------------------------------------------ |
| Dataset name | Unchanged |
| Column configuration | Unchanged — verify headers still align |
| Entitlements | Unchanged |
| Translation mappings | Resolve against the new file on next query |
| Previous file (replace) | Permanently removed |
## Troubleshooting
**Upload/Replace not visible** — Confirm Admin+ role and File Upload connection type.
**Validation error** — Check delimiter, encoding, and that format matches (`.csv` for `.csv`).
**No data after upload** — Column headers or date/key fields may not match the file; re-onboard if structure changed.
**Stale data after replace** — Refresh pre-generated cache if enabled.
# Fine-Grained Permissions
Source: https://knowledge.cloudquant.com/administration/fine-grained-permissions
Restrict user query scope with trial periods, date ranges, row limits, keys, and column filters
# Fine-grained permissions
Fine-grained permissions let Super Admins apply additional restrictions on top of standard dataset entitlements. Use them to enforce trial periods, limit historical lookback, cap query volume, or restrict which keys and columns a user can access.
Fine-grained rules apply **per dataset** or **per dataset group** and are configured from the user's **Data Permissions** dialog.
## When to use fine-grained permissions
| Scenario | Recommended restriction |
| ----------------------------------- | --------------------------------------- |
| Time-limited trial | **Trial Range** |
| Historical data only (no live feed) | **Data Query Range** → Historical Only |
| Rolling window (e.g., last 90 days) | **Data Query Range** → Rolling Window |
| Fixed evaluation period | **Data Query Range** → Fixed Date Range |
| Cap the width of a time-range query | **Max Query Days** |
| Cap download size | **Max Rows** |
| Limit API usage | **Max Queries** |
| Restrict to a symbol universe | **Keys** |
| Hide sensitive columns | **Columns** |
Fine-grained permissions require an underlying dataset or group entitlement. They cannot grant access on their own — the user must already be entitled to the dataset or group.
## Opening fine-grained settings
From the Users page, click the data-permissions icon for the target user.
Ensure the dataset or dataset group row is checked in the permissions table.
Click the gear icon (**Actions** column) on the selected row. The icon appears filled (blue) when fine-grained rules already exist for that item.
Enable the restriction sections you need, set values, click **Done** to return to the permissions table, then **Save Changes**.
Fine-grained settings are not saved until you click **Save Changes** on the main Data Permissions dialog. Clicking **Done** only returns you to the entitlement table.
## Restriction sections
Each section is controlled by an **Enable** toggle. Disabled sections are ignored — their fields are not sent to the backend.
### Trial Range
Limits when the user can access the platform for a specific dataset or group, independent of the data date range.
| Field | Description |
| ---------------- | ---------------------------------------- |
| **Trial Period** | Start and end dates for the trial window |
Outside the trial window, queries against the restricted dataset or group are denied even if other entitlements exist.
### Data Query Range
Controls what type of data the user can query and over what time horizon.
| Type | Behavior |
| -------------------------- | -------------------------------------------------------------------------------- |
| **Historical Only** | Live/real-time data is blocked; only historical queries are allowed |
| **Live Only** | Only live/real-time data is accessible |
| **Both Historical & Live** | No restriction on query mode |
| **Fixed Date Range** | Queries are limited to a specific start/end date range |
| **Rolling Window** | Queries are limited to the last N days from the current date (**Days Lookback**) |
### Limits and Settings
Caps query scope and restricts the symbol universe and visible columns.
| Field | Description |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Max Rows** | Maximum number of rows returned per query |
| **Max Query Days** (2.4+) | Maximum number of calendar days a single query's `back_to`–`as_of` window may span. Use this on high-volume datasets so one oversized request cannot stall other users. |
| **Max Queries** | Maximum number of queries allowed (usage cap) |
| **Max Keys** | Maximum number of keys (symbols) that can be queried in a single request |
| **Keys** | Allow-list of specific keys. Enter individually or paste comma-separated values |
| **Columns** | Allow-list of column names from the dataset schema. Search and multi-select from available columns |
System columns (`_seq`, `timestamp`, `muts`, `symbol`) are excluded from the column picker because they are always required for query execution.
## Dataset vs group rules
Fine-grained permissions can target either:
* **A single dataset** — restrictions apply only to that dataset
* **A dataset group** — restrictions apply to the group entitlement as a whole
Group-level rules are configured the same way: select the group row, click the gear icon, and set restrictions. The dialog title shows the group name.
## Removing fine-grained permissions
To remove all restrictions for a dataset or group:
1. Open the fine-grained settings (gear icon).
2. Disable all three sections (**Trial Range**, **Data Query Range**, **Limits and Settings**), or clear all restriction values.
3. Click **Done**, then **Save Changes**.
Alternatively, revoke the entire dataset or group entitlement — removing an entitlement automatically removes its associated fine-grained rules.
## Interaction with connection entitlements
Datasets inherited from a **connection-level** entitlement cannot have individual fine-grained rules configured in the per-user permissions UI. The gear icon is hidden for locked (inherited) datasets.
To apply fine-grained restrictions in this case, either:
* Assign a direct dataset entitlement instead of (or in addition to) the connection entitlement, or
* Manage access at the connection level through connection permission settings
## How restrictions are enforced
When a user submits a query, the entitlements service evaluates:
1. Whether the user has a dataset, group, or connection entitlement
2. Whether any fine-grained rules apply to that dataset
3. Whether the query parameters (date range, day span, keys, columns, row count) fall within the allowed bounds
If a query violates a restriction, it is rejected before data is returned. In Liberator **2.4** and later, the error names the limit that was exceeded (for example max rows, max query days, or missing entitlement) and the configured value. Callers can correct the request instead of treating every rejection as "not entitled."
## Example configurations
### 90-day rolling trial
1. Enable **Trial Range** and set the trial period dates.
2. Enable **Data Query Range**, select **Rolling Window**, set **Days Lookback** to `90`.
3. Save.
### Cap a single query's date span (2.4+)
1. Enable **Limits and Settings**.
2. Set **Max Query Days** to the widest window you will allow (for example `7` or `30`).
3. Optionally set **Max Rows** as a second backstop.
4. Save.
### Symbol-restricted research access
1. Enable **Limits and Settings**.
2. Set **Keys** to the allowed symbol list (e.g., `AAPL`, `MSFT`, `GOOG`).
3. Optionally set **Max Rows** to cap result size.
4. Save.
### Column-redacted dataset
1. Enable **Limits and Settings**.
2. Use **Columns** to select only the fields the user should see.
3. Save.
## Checking effective access
Users can verify entitled datasets from Python:
```python theme={null}
liberator.datasets(entitled=True)
```
Fine-grained restrictions are enforced at query time and are not visible in the dataset list. If a query fails due to a restriction, the error message indicates which limit was exceeded and, in Liberator **2.4** and later, the configured cap.
For admin-side verification, open the user's **View** dialog or the **JSON** tab (Super Admin) to inspect the full permissions payload including `fine_grain_permissions`.
# Managing Users & Entitlements
Source: https://knowledge.cloudquant.com/administration/managing-users-and-entitlements
Add, create, deactivate users and grant or revoke dataset entitlements in the CloudQuant Data Liberator admin portal
# Managing users and entitlements
This guide covers the full user lifecycle in the CloudQuant Data Liberator admin portal — from onboarding a new account to assigning and revoking dataset access.
## Prerequisites
* **Super Admin** role for creating users, editing details, assigning entitlements, and deactivating users
* **Admin** or **Super Admin** role to add existing user accounts to the permissions system
Navigate to **Users** in the admin portal sidebar.
## User lifecycle
```mermaid theme={null}
flowchart LR
A[Create or provision account] --> B[Add to Liberator]
B --> C[Assign entitlements]
C --> D[Optional fine-grained rules]
D --> E[User queries data]
E --> F[Deactivate when offboarded]
F --> G[Re-activate if needed]
```
## Creating a new user
Super Admins can create a brand-new account directly from the Users page.
Click **Create User** in the top-right of the Users page.
Fill in the required fields:
* **Username** — unique login identifier
* **Email**
* **First name** and **Last name**
* **Company** — select an existing company or type a new one
* **Password** — minimum 8 characters; use the generate button for a secure password (not required when using invite-via-email)
Click **Create**. The platform account is created.
In Liberator **2.4** and later, create and enroll happen in **one step**. The new user is added to Liberator permissions as part of Create User, so you do not need a separate **Add Users** pass for accounts you just created. Use **Add Users** only for existing platform accounts that are not yet in Liberator.
## Invite via email
In Liberator **2.3** and later, Super Admins can provision users without sharing temporary passwords out of band.
Open **Create User** and turn on **Invite via email**.
Provide username, email, name, and company as usual. Liberator sends the invite to the email address you enter.
The recipient opens the invite link, verifies their email, and sets their own password. Liberator requires a password change on first login for newly invited accounts.
In Liberator **2.4** and later, invite and email-verification messages include the username the recipient should sign in with.
### Send a password-reset email
From the Users table **Actions** menu, select **Send reset email** to trigger a self-service password reset for an existing account. Use this when a user cannot sign in and you do not want to set a temporary password yourself.
## Adding users to Liberator
Before a user can receive dataset entitlements, their account must exist in the Liberator permissions system. Use **Add Users** to enroll accounts that are not yet active in Liberator.
Click **Add Users** on the Users page.
The **Available Users** tab lists enabled accounts that are not yet in the permissions system. Use the column filters to search by name, email, or (in Liberator **2.4** and later) **company**, select one or more users, then click **Add**.
The **Deactivated Users** tab lists previously deactivated accounts. Select users and click **Re-activate** to restore their Liberator enrollment. You will still need to re-assign entitlements if they were removed during deactivation.
You can add multiple users in a single operation. Selected users appear as pills above the table for easy review before confirming.
## Editing user details
Super Admins can update profile information and platform roles from the Users table.
In the **Actions** column, click the edit icon for the target user.
Modify any of:
* First name, last name, email, company
* **User Role** — `User`, `Admin`, or `Super Admin` (you can only assign roles at or below your own level)
* **Password** — optionally set a temporary password (copied to clipboard on save)
Click **Save Changes**.
## Assigning dataset entitlements
Entitlements control which datasets a user can query. Super Admins manage entitlements through the **Data Permissions** dialog.
In the Users table **Actions** column, click the data-permissions icon for the target user.
The permissions table lists all datasets and dataset groups. Check the rows you want to grant:
* **Dataset** rows grant access to a single dataset
* **Group** rows grant access to every dataset in that dataset group
Selected items appear as pills above the table. Remove a pill to revoke that entitlement.
Click **Save Changes** and confirm. Changes take effect immediately for the user's next query.
### Inherited (locked) datasets
If a user has **connection-level** entitlements, all datasets in that connection are automatically included. These datasets:
* Appear with a lock icon in the permissions table
* Show as **(locked)** on their selection pill
* Cannot be individually deselected until the connection entitlement is removed
Connection entitlements are typically managed through connection-level permission settings rather than the per-user dataset table.
### Viewing a user's access
Click the **View** action on any user to open a read-only summary of their connections, datasets, groups, and role. Super Admins also see a **JSON** tab with the raw permissions payload.
## Removing entitlements
To revoke dataset or group access:
1. Open **Data Permissions** for the user.
2. Deselect the dataset or group row in the table, or click the remove button on its pill.
3. Click **Save Changes**.
Removing a dataset entitlement also removes any fine-grained permission rules attached to that dataset. See [Fine-Grained Permissions](/administration/fine-grained-permissions) for details.
Deselecting a dataset group removes the group entitlement but does not affect individually assigned datasets within that group unless you deselect those rows separately.
## Deactivating users
Deactivation removes a user from the active Liberator permissions system. The user account is disabled and the user's permissions row is deleted on the backend.
### Single user
Click the delete icon in the **Actions** column, or open **Data Permissions** and click the trash icon in the modal header.
Confirm the deactivation. The user moves to the **Deactivated Users** tab in the Add Users dialog.
### Bulk deactivation
Super Admins can deactivate multiple users at once:
1. Select users using the checkboxes in the rightmost column.
2. Click **Deactivate Selected**.
3. Type `DEACTIVATE` to confirm.
You cannot deactivate your own account. Bulk deactivation requires Super Admin role.
## Re-activating users
1. Click **Add Users** on the Users page.
2. Switch to the **Deactivated Users** tab.
3. Select the users to restore and click **Re-activate**.
4. Re-assign any dataset entitlements and fine-grained rules that were removed during deactivation.
## Verifying entitlements (end user)
Users can confirm their access from Python:
```python theme={null}
# Datasets the user is entitled to query
liberator.datasets(entitled=True)
# All datasets (including those without access)
liberator.datasets(entitled=False)
```
See [Checking Dataset Access](/python-guide/dataset-access) for schema and detail commands.
## Common tasks
| Task | Where | Required role |
| --------------------------------- | ---------------------------------------------- | -------------------- |
| Create a new account | Users → **Create User** | Super Admin |
| Combined create and enroll (2.4+) | Users → **Create User** | Super Admin |
| Invite via email (2.3+) | Users → **Create User** → **Invite via email** | Super Admin |
| Send password-reset email (2.3+) | Users → **Send reset email** | Super Admin |
| Enroll an existing account | Users → **Add Users** | Admin or Super Admin |
| Grant dataset access | Users → **Data Permissions** | Super Admin |
| Revoke dataset access | Users → **Data Permissions** → deselect | Super Admin |
| Change platform role | Users → **Edit** | Super Admin |
| Deactivate a user | Users → delete action | Super Admin |
| Re-activate a user | Users → **Add Users** → Deactivated tab | Admin or Super Admin |
# Administration Overview
Source: https://knowledge.cloudquant.com/administration/overview
Manage users, dataset entitlements, and fine-grained access controls in the CloudQuant Data Liberator admin portal
# Administration overview
The CloudQuant Data Liberator admin portal lets organization administrators control who can access the platform and which datasets each user can query. Access is managed through **entitlements** — explicit grants to datasets, dataset groups, and connections — with optional **fine-grained permissions** that restrict query scope.
## Who can manage access
The admin portal uses a role hierarchy. Each role can assign roles at or below its own level.
| Role | Description |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **User** | Query entitled datasets. No admin portal access. |
| **Admin** | Manage connections, datasets, and user groups. Can add existing user accounts to the permissions system. |
| **Super Admin** | Full platform administration — create users, edit roles, assign entitlements, configure fine-grained permissions, and deactivate users. |
Most user and entitlement management actions described in this section require **Super Admin**. Admins can view the Users page and add users to Liberator, but editing user details, assigning dataset entitlements, and deactivating users are Super Admin actions.
## Permission model
Access is layered:
1. **Platform role** (`user`, `admin`, `super_admin`) — controls what a person can do in the admin portal.
2. **Connection entitlements** — grant access to all datasets within a connection. Datasets inherited from a connection appear as locked in the permissions UI.
3. **Dataset entitlements** — grant access to individual datasets at **Editor** or **User** level.
4. **Dataset group entitlements** — grant access to a curated bundle of datasets defined in **Dataset Groups**.
5. **Fine-grained permissions** — optional per-dataset or per-group restrictions on date range, query day span, query type, row limits, keys, and columns.
```mermaid theme={null}
flowchart TD
A[User account] --> B[Platform role]
A --> C[Connection entitlements]
A --> D[Dataset entitlements]
A --> E[Dataset group entitlements]
D --> F[Fine-grained permissions]
E --> F
C --> G[Inherited dataset access]
```
### Editor vs user
For datasets and connections, entitlements are assigned at one of two levels:
| Level | Meaning |
| ---------- | ---------------------------------------------------------------------------------------------------------------------- |
| **Editor** | Full access to query the dataset (or all datasets in a connection). |
| **User** | Query access at the usage level. In the UI this is labeled **User**; internally it maps to the usage entitlement tier. |
When a user has connection-level access, all datasets in that connection are inherited automatically. Those datasets cannot be individually removed until the connection entitlement is revoked.
## Admin portal navigation
Super Admins and Admins see these relevant sections in the sidebar:
| Page | Purpose |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Users** | Add users, assign entitlements, configure fine-grained permissions |
| **User Groups** | Manage groups of users for bulk entitlement assignment |
| **Dataset Groups** | Manage bundles of datasets for bulk entitlement assignment |
| **Connections** | Configure data source connections |
| **Datasets** | View and manage individual datasets. In Liberator **2.4** and later, catalog cards show date range, symbol count, status, and last updated. Administrators can configure how often symbol discovery refreshes. |
## Related guides
Describe a source in plain language and auto-fill the Create Dataset wizard
Natural-language usage, user, and entitlement reports in the Admin UI
Super-Admin per-query usage dataset (`usage_log`) via the standard query API
Add, create, deactivate users and grant or revoke dataset access
Restrict query date ranges, row limits, keys, and columns per user
Public vs private datasets and the Publicly Available toggle
Schedule warm Parquet caches in S3 or GCS before users query
Lookup-table enrichment with translate=true
Authentication, network security, and the end-user view of entitlements
How users verify their entitled datasets from Python
# Dataset Metadata
Source: https://knowledge.cloudquant.com/api-reference/concepts/dataset-metadata
Fetch concordance-backed row counts, symbol counts, and date spans with liberator.metadata() (POST /liberator/metadata).
# Dataset metadata
Use `liberator.metadata()` to fetch **concordance-backed metrics** for one or more datasets — row counts, symbol counts, and start/end timestamps — without downloading source rows.
This is a Liberator **2.3+** API. It calls `POST /liberator/metadata` and is separate from `liberator.query()` / `liberator.get_dataframe()`. Do not pass `summary` or `symbol` to `query`; those parameters belong only on `metadata`.
The Python SDK helper ships with Liberator 2.3 client packages. Download the latest `liberator.py` from your Liberator Profile page if your client does not yet expose `liberator.metadata`.
## Parameters
| Parameter | Description | Type | Default |
| --------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------- | --------------------------- |
| `name` | Dataset name, or a list of dataset names | String or list | — (required) |
| `summary` | Collapse the per-day dimension into a whole-history summary | Boolean | `false` |
| `symbol` | Return per-symbol grain instead of per-dataset totals | Boolean | `false` |
| `symbols` | Restrict metrics to this symbol or list of symbols (also used for entitlement checks) | String or list | `null` |
| `back_to` | Beginning of the data-time window | Date string, microseconds UTC, or `null` (no lower bound) | `null` |
| `as_of` | End of the data-time window | Date string, microseconds UTC, or `null` (now) | `null` |
| `user` | Your CloudQuant-assigned user identifier | String | from `liberator.json` / env |
| `token` | Your authentication token | String | from `liberator.json` / env |
## Grain matrix (`summary` × `symbol`)
| `summary` | `symbol` | Result grain |
| --------- | -------- | ------------------------- |
| `false` | `false` | Per day, dataset totals |
| `false` | `true` | Per day, per symbol |
| `true` | `false` | Whole-dataset totals |
| `true` | `true` | Whole-history, per symbol |
## Basic example
```python theme={null}
import liberator
liberator.url = "https://api.cloudquant.ai"
metrics = liberator.metadata(
name="daily_bars",
summary=False,
symbol=False,
)
# list of metric dicts, for example:
# [{"dataset": "...", "date": "...", "symbol_count": ..., "row_count": ..., "start": ..., "end": ...}, ...]
print(metrics)
```
## Whole-dataset summary
```python theme={null}
metrics = liberator.metadata(
name="daily_bars",
summary=True,
symbol=False,
)
```
## Per-symbol metrics
```python theme={null}
# Per day + per symbol
daily_by_symbol = liberator.metadata(
name="daily_bars",
summary=False,
symbol=True,
symbols=["AAPL", "MSFT"],
)
# Whole-history per symbol
history_by_symbol = liberator.metadata(
name="daily_bars",
summary=True,
symbol=True,
symbols=["AAPL"],
)
```
## Multiple datasets
```python theme={null}
metrics = liberator.metadata(
name=["daily_bars", "minute_bars"],
summary=True,
symbol=False,
)
```
## Windowed metadata
```python theme={null}
metrics = liberator.metadata(
name="daily_bars",
summary=False,
symbol=False,
back_to="2025-01-01",
as_of="2025-01-31",
)
```
## Return shape
`liberator.metadata()` returns a **list of metric dictionaries** (the `metrics` array from the streamed response). Typical fields include:
| Field | Meaning |
| --------------- | ----------------------------------------------- |
| `dataset` | Dataset name |
| `date` | Calendar day (present when `summary=False`) |
| `symbol` | Symbol key (present when `symbol=True`) |
| `symbol_count` | Distinct symbols in the grain (when applicable) |
| `row_count` | Row count for the grain |
| `start` / `end` | Timestamp bounds for the grain |
The HTTP response is streamed NDJSON (heartbeat frames, then a final `{"metrics": [...]}` object). The Python helper consumes heartbeats and returns only the metrics list.
## Related
* [Listing datasets](/api-reference/concepts/listing-datasets) — catalog, schema, and entitlement discovery via `liberator.datasets`
* [Query parameters](/api-reference/concepts/query-parameters) — row-level queries via `liberator.query` (not metadata)
* [Checking dataset access](/python-guide/dataset-access) — Python guide for exploring entitled datasets
# Example Queries
Source: https://knowledge.cloudquant.com/api-reference/concepts/example-queries
Practical examples of CloudQuant Data Liberator queries including last known value, time series, all symbols, live datasets, and stats.
# Example queries
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `--cert-type P12 --cert liberator.pfx` flags shown in the curl examples below are only needed for releases prior to 2.0.
## Last known value
A query without date parameters retrieves the most recent available data.
```python Python theme={null}
import liberator
res = liberator.query(name='daily_bars')
df = liberator.get_dataframe(res)
df[['symbol', 'timestamp', 'open', 'high', 'low', 'close', 'volume']]
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"}
});
var df = liberator.get_dataframe(res);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
```
```javascript JavaScript theme={null}
let params = {
symbols: "AAPL",
name: "daily_bars"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Liberator liberator = new Liberator();
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
}});
for (Liberator.QueryResult res : liberator.Generate()) {
Object data = res.GetData();
if (data instanceof org.apache.arrow.vector.VectorSchemaRoot) {
printRecordBatch((org.apache.arrow.vector.VectorSchemaRoot) data);
} else if (data instanceof javax.json.JsonValue) {
System.out.println((javax.json.JsonValue) data);
}
}
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":["AAPL","IBM"],"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","debug_stream":false}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"name", "daily_bars"s}
}));
(void)arrow::PrettyPrint(*table, arrow::PrettyPrintOptions(0, 1), &std::cout);
```
Rather than displaying the entire result, you can select specific columns or pass a `fields` parameter to include only the data you need. Some key fields are returned automatically.
## Time series queries
Two date parameters control historical data retrieval:
* **`as_of`** — Retrieve data from any past date showing information as known on that date. Defaults to current time. Format: `YYYY-MM-DD HH:MM:SS` (time optional).
* **`back_to`** — Specifies where the returned dataset begins. Reads all data "back to" the specified date. Format: `YYYY-MM-DD HH:MM:SS` (time optional).
If `as_of` and `back_to` are identical, the result will not be a time series.
```python Python theme={null}
df = liberator.get_dataframe(liberator.query(
name='daily_bars',
as_of='2020-11-15',
symbols=['FB', 'AAPL', 'NFLX', 'GOOG', 'MSFT', 'IBM']))
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
}
```
```javascript JavaScript theme={null}
let params = {
back_to: "2021-01-12 00:00:00",
as_of: "2021-01-12 23:59:59",
symbols: "AAPL",
name: "daily_bars"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("back_to", "2020-01-01");
put("symbols", "AAPL");
}});
for (Liberator.QueryResult res : liberator.Generate()) {
Object data = res.GetData();
if (data instanceof org.apache.arrow.vector.VectorSchemaRoot) {
printRecordBatch((org.apache.arrow.vector.VectorSchemaRoot) data);
} else if (data instanceof javax.json.JsonValue) {
System.out.println((javax.json.JsonValue) data);
}
}
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
symbols="AAPL",
as_of="2019-09-15",
back_to="2019-09-10"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","back_to":"2022-12-01","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","debug_stream":false}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
{"as_of", "2021-04-10"s},
{"back_to", "2021-03-10"s}
}));
```
## Query all symbols
Omit the `symbols` parameter to retrieve all symbols in a dataset.
```python Python theme={null}
df = liberator.get_dataframe(liberator.query(
name='daily_bars',
as_of='2020-11-15'))
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
}});
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"name":"daily_bars","as_of":"2020-11-15","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query
```
## Query a live dataset
Set the `as_of` parameter to `"live"` for real-time data streaming.
```python Python theme={null}
res = liberator.query(name='live_nasdaq_composite', as_of='live')
for batch in res:
df = batch.to_pandas()
print(df)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "live_nasdaq_composite"},
{"as_of", "live"},
{"symbols", "AAPL"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
}
```
```cpp C++ theme={null}
Liberator::Result ptr = liberator.query({
{"symbols", std::vector{"AAPL"}},
{"name", "live_nasdaq_composite"s},
{"as_of", "live"s}
});
auto generator = *std::get_if(&ptr);
for (auto res = generator(); res.index(); res = generator())
{
auto batch = *std::get_if<2>(&res);
(void)arrow::PrettyPrint(*batch, arrow::PrettyPrintOptions(0, 1), &std::cout);
}
```
## Query stats as JSON
Setting the `stats` parameter to `"total"` returns a JSON result with the count per symbol instead of data rows.
```python Python theme={null}
res = liberator.query(
name='daily_bars',
as_of='2021-04-10',
back_to='2021-03-10',
symbols=['ES'],
stats='total'
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"},
{"stats", "total"}
});
Console.WriteLine(JsonDocumentToIndentedString(res));
```
```r R theme={null}
print(liberator::query(
symbols="AAPL",
name="daily_bars",
as_of="2019-09-15",
back_to="2019-09-10",
stats="total"
))
```
## Get DataFrame / table
Convert query results into a tabular format native to your language.
```python Python theme={null}
df = liberator.get_dataframe(
liberator.query(name='daily_bars', as_of='2020-11-15', symbols=['AAPL'])
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
var df = liberator.get_dataframe(res);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
```
```r R theme={null}
res = liberator::get_dataframe(liberator::query(
symbols="AAPL",
name="daily_bars",
as_of="2025-01-01",
back_to="2024-12-01"
))
print(res)
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
}));
(void)arrow::PrettyPrint(*table, arrow::PrettyPrintOptions(0, 1), &std::cout);
```
# Listing Datasets
Source: https://knowledge.cloudquant.com/api-reference/concepts/listing-datasets
How to list, filter, and inspect available datasets using the CloudQuant Data Liberator API across all supported languages.
# Listing datasets
The `datasets` function retrieves a JSON-formatted list of all datasets available in CloudQuant Data Liberator, with optional filtering and metadata.
## Parameters
| Parameter | Description | Type | Default |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------- |
| `entitled` | Only retrieve datasets you have access to | Boolean | `false` |
| `schema` | Include dataset descriptions and metadata | Boolean | `false` |
| `details` | Include column-level information (names, descriptions). In Liberator **2.4+**, also includes catalog-card fields: date range, symbol count, status, and last updated. Requires `schema` to also be set. | Boolean | `false` |
| `user` | Your CloudQuant-assigned user identifier | String | — |
| `token` | Your assigned authentication token | String | — |
## List all datasets
```python Python theme={null}
import liberator
res = liberator.datasets()
print(res)
```
```csharp C# theme={null}
Liberator liberator;
var res = liberator.datasets();
Console.WriteLine(JsonDocumentToIndentedString(res));
```
```javascript JavaScript theme={null}
let dataset_params = {
schema: false,
details: true
};
liberator.datasets(dataset_params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Liberator liberator = new Liberator();
Object datasets_id = liberator.datasets(new HashMap<>());
for (Liberator.QueryResult res : liberator.Generate()) {
Object data = res.GetData();
if (data instanceof javax.json.JsonValue) {
System.out.println((javax.json.JsonValue) data);
}
}
```
```r R theme={null}
print(liberator::datasets())
```
```cpp C++ theme={null}
Liberator liberator;
auto ptr = liberator.datasets({});
auto json = *std::get_if<0>(&ptr);
if (json)
{
rapidjson::StringBuffer buf;
rapidjson::Writer writer(buf);
json->Accept(writer);
std::cout << buf.GetString() << std::endl;
}
```
### Example output
```json theme={null}
{
"Intraday": ["SSR_Forward_DSI", "Halt_Production_DSI", "SSR_Production_DSI", "News", "Twitter", "Stocktwits"],
"Monthly": ["Joblink Ticker Mapping", "VerticalKnowledge Indeed"],
"Market Data": ["daily_bars", "minute_bars", "nbbo", "trades", "daily_bars_adjusted"]
}
```
## List entitled datasets only
Filter results to show only datasets matching your access permissions.
```python Python theme={null}
liberator.datasets(entitled=True)
```
```csharp C# theme={null}
var res = liberator.datasets(new Dictionary() {
{"entitled", true}
});
```
```java Java theme={null}
Object datasets_id = liberator.datasets(new HashMap<>() {{
put("entitled", true);
}});
```
```r R theme={null}
print(liberator::datasets(entitled=TRUE))
```
```cpp C++ theme={null}
liberator.datasets({{"entitled", true}});
```
## Include schema and column details
```python Python theme={null}
# Schema only
liberator.datasets(schema=True)
# Schema with column details (2.4+: also date range, symbol count, status, last updated)
liberator.datasets(schema=True, details=True)
```
```csharp C# theme={null}
var res = liberator.datasets(new Dictionary() {
{"schema", true},
{"details", true}
});
```
```r R theme={null}
print(liberator::datasets(schema=TRUE, details=TRUE))
```
```cpp C++ theme={null}
liberator.datasets({{"schema", true}, {"details", true}});
```
## Related
For concordance-backed row counts, symbol counts, and date spans without downloading source rows, see [Dataset metadata](/api-reference/concepts/dataset-metadata) (`liberator.metadata`, Liberator 2.3+).
In Liberator **2.4** and later, `liberator.datasets(schema=True, details=True)` also returns catalog-card health (date range covered, symbol count, status, last updated) so you can check freshness before you query. The same fields appear on dataset cards in the Liberator catalog UI.
# Queries & Working with Large Datasets
Source: https://knowledge.cloudquant.com/api-reference/concepts/queries-large-datasets
Best practices for querying large datasets with the CloudQuant Data Liberator API, including point-in-time and time series query patterns.
# Queries & working with large datasets
The CloudQuant Data Liberator service frequently returns extensive result sets, sometimes reaching millions of rows. These queries can be time-consuming to execute. Since data frequencies vary across datasets, start with narrow timeframes for specific symbols before expanding scope.
Consider running a point-in-time query for one symbol to get an idea of how large your dataset is prior to running other queries.
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `--cert-type P12 --cert liberator.pfx` flags shown in the curl examples below are only needed for releases prior to 2.0.
## Query types
Most dataset queries fall into two categories:
* **Point-in-time queries**
* **Time series queries**
Familiarity with both query types early on will improve your API usage efficiency.
## Point-in-time queries
Omit the `back_to` parameter to receive single point-in-time data for each specified symbol based on the `as_of` date/time. If you also exclude `as_of`, CloudQuant Data Liberator defaults to the current date/time.
```python Python theme={null}
df = liberator.get_dataframe(
liberator.query(
name='daily_bars',
as_of='2020-11-15',
symbols=['AAPL']
)
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2020-11-15"},
{"symbols", "AAPL"}
});
```
```javascript JavaScript theme={null}
let params = {
symbols: "AAPL",
name: "daily_bars",
as_of: "2020-11-15"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("symbols", "AAPL");
}});
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
as_of="2020-11-15",
symbols="AAPL"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","as_of":"2020-11-15","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL"}},
{"name", "daily_bars"s},
{"as_of", "2020-11-15"s}
}));
```
## Time series queries
Include a `back_to` parameter that precedes your `as_of` value. When `as_of` is excluded, it defaults to the present moment.
```python Python theme={null}
df = liberator.get_dataframe(
liberator.query(
name='daily_bars',
as_of='2020-11-15',
back_to='2018-11-15',
symbols=['FB', 'AAPL', 'NFLX', 'GOOG', 'MSFT', 'IBM']
)
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
}
```
```javascript JavaScript theme={null}
let params = {
back_to: "2021-01-12 00:00:00",
as_of: "2021-01-12 23:59:59",
symbols: "AAPL",
name: "daily_bars"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("back_to", "2020-01-01");
put("symbols", "AAPL");
}});
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
symbols="AAPL",
as_of="2019-09-15",
back_to="2019-09-10"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","back_to":"2022-12-01","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
{"as_of", "2021-04-10"s},
{"back_to", "2021-03-10"s}
}));
```
When using `back_to` or `as_of` parameters, the time component is always used even if you do not specify it. Therefore, if you say `as_of: "2023-01-15"`, you are actually saying `as_of: "2023-01-15 00:00:00"`. This may affect result precision depending on your data requirements.
# Query Parameters Reference
Source: https://knowledge.cloudquant.com/api-reference/concepts/query-parameters
Complete reference for all parameters accepted by the CloudQuant Data Liberator query function across all SDKs.
# Query parameters reference
This page documents the parameters accepted by the `query` function across all CloudQuant Data Liberator SDKs. Parameter names are consistent across languages; only the types and calling conventions differ.
At least one of the following is required for every query:
* A dataset via `name` **or** a raw `sql` statement.
* A time anchor: `back_to`, `as_of`, `max_lookback` (LKV mode), or `record_limit` (N-Query mode).
Unknown parameters are rejected by the server when strict validation is enabled (the default).
## Time range parameters
| Parameter | Default | Description |
| -------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `as_of` | `None` | End time for the query. `None` means "now". Integer values are interpreted as microseconds since epoch; strings are interpreted as datetimes. In N-Query backward mode the boundary is **exclusive** (`muts < as_of`). |
| `back_to` | `None` | Start time for the query. `None` means as-of only (single point in time, or LKV). In N-Query forward mode the boundary is **inclusive** (`muts >= back_to`). |
| `max_lookback` | (from dataset config) | Used for LKV (last-known-value) queries. Integer values **less than 1,000,000** are treated as a number of partitions to look back; values **greater than or equal to 1,000,000** are treated as a duration in microseconds. May be supplied per-query or inherited from dataset config. |
For LKV mode, omit `back_to` and use `as_of` (or the default "now") together with `max_lookback`. For N-Query mode, provide either `as_of` (backward) or `back_to` (forward) along with `record_limit`.
### Date format
Datetime strings use the format `YYYY-MM-DD HH:MM:SS` (the time portion is optional). When the time portion is omitted, `00:00:00` is assumed — for example, `as_of: "2023-01-15"` is interpreted as `as_of: "2023-01-15 00:00:00"`. If `as_of` and `back_to` are identical, the result is a single point rather than a time series.
## Dataset & symbol parameters
| Parameter | Default | Description |
| ----------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `None` | Dataset name(s). A **string** selects a single dataset; a **list of strings** triggers a superquery (multi-dataset merge). Required unless `sql` is provided. |
| `symbols` | `None` | Symbol(s) to filter on. Accepts a string, a list of strings, or `None` for all symbols in the dataset. In Liberator **2.4+**, an unescaped `%` in a value is a `LIKE` wildcard (for example `'PWR%'`). Mix exact keys and patterns in one list. Escape a literal percent as `\%`. |
| `keys` | `None` | Synonym for `symbols`. |
| `fields` | `None` | Field names to return. List of strings or `None` for all fields. Mandatory fields cannot be removed. |
| `distinct` | `["symbol"]` | Partition key for LKV queries: the columns used in `PARTITION BY` (for example, `["symbol"]` or `["symbol", "dataset"]`). |
| `translate` | `None` | When set to `true` or `false`, enables or disables [dataset field mapping](/administration/dataset-field-mapping) lookups configured on the dataset. |
| `where` | `None` | Optional SQL predicate appended to generated filters with `AND`. Omit the `WHERE` keyword — supply only the condition (e.g. `"volume > 1000000"`). Not compatible with `sql`. |
| `sql` | `None` | Raw SQL `SELECT` statement. When present, replaces parameter-based query construction (`name`, time range, `where`, etc.). See [Raw SQL](#raw-sql) below. |
## Custom WHERE filters
The `where` parameter adds extra predicates on top of the time-range and symbol filters CloudQuant Data Liberator generates automatically. The value is appended as `AND ()` to the internal query.
| Aspect | Detail |
| --------------------- | ---------------------------------------------------------------------------------------------- |
| **Syntax** | SQL expression only — no leading `WHERE` |
| **Compatible with** | `name`, `back_to`, `as_of`, `symbols`, `fields`, `record_limit`, LKV (`max_lookback`), N-Query |
| **Incompatible with** | `sql` |
| **Column quoting** | Column names matching the dataset schema are auto-quoted when required |
```python Python theme={null}
result = liberator.query(
name="daily_bars",
symbols=["AAPL"],
back_to="2024-01-01",
as_of="2024-07-01",
where="volume > 1000000"
)
```
```csharp C# theme={null}
var result = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"symbols", new[] {"AAPL"}},
{"back_to", "2024-01-01"},
{"as_of", "2024-07-01"},
{"where", "volume > 1000000"}
});
```
```r R theme={null}
result <- liberator::query(
name = "daily_bars",
symbols = c("AAPL"),
back_to = "2024-01-01",
as_of = "2024-07-01",
where = "volume > 1000000"
)
```
For multi-key datasets (e.g. concordance tables with a `dataset` column), `where` can filter on columns beyond `symbol`:
```python theme={null}
result = liberator.query(
name="concordance",
back_to="2024-01-01",
as_of="2025-01-01",
where="dataset = 'my_derived_dataset'"
)
```
## Raw SQL
The `sql` parameter accepts a complete SQL `SELECT` and bypasses automatic query construction. Use it when you need direct control over joins, subqueries, or complex predicates.
### Requirements
| Rule | Detail |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Statement type | Must be a `SELECT` (each arm of `UNION` / `INTERSECT` / `EXCEPT` is validated separately) |
| `WHERE` clause | Required on every `SELECT` arm |
| Time-range predicate | By default, `WHERE` must include `=`, `>`, `>=`, `<`, `<=`, or `BETWEEN` on `muts` or columns listed in `LIBERATOR_SQL_TIME_RANGE_COLUMNS` |
| Dataset references | Quote table names with double quotes: `"my_dataset"` |
| String literals | Use `$$value$$` dollar-quoting for symbol and string filters |
| Timestamps | Express `muts` bounds as microsecond integers |
### Parameters allowed with `sql`
When `sql` is supplied, only auth, transfer, and cache options may accompany it. Non-empty construction parameters (`name`, `back_to`, `as_of`, `where`, `fields`, etc.) raise an error.
Allowed alongside `sql`: `user`, `system`, `token`, `compress`, `json_xfer`, `batch_size`, `debug_stream`, `force_regen`, `dependent_partition_name`, `skip_validation`, `skip_discovery`, `symbols`, `symbol_key_sequences`, `data_key_column`.
Do not combine `sql` with `name`, `where`, `back_to`, `as_of`, or other query-construction parameters. Choose parameter-based queries or raw SQL, not both.
## Query limits & ordering
| Parameter | Default | Description |
| -------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `record_limit` | `None` | When set to a non-zero integer, triggers **N-Query mode**. **Negative** values (e.g. `-200`) apply a per-symbol limit. **Positive** values (e.g. `200`) apply a global clamp (most recent N records across all symbols). Ignored when both `as_of` and `back_to` are provided. |
| `order` | `None` | Result ordering. One of `"asc_strict"`, `"asc_loose"`, `"desc_strict"`, `"desc_loose"`. Default is ascending. |
### N-query mode
N-Query mode is activated by supplying a non-zero `record_limit`.
* **Anchor:** Backward N-Query uses `as_of` (exclusive); forward N-Query uses `back_to` (inclusive). Exactly one anchor is required.
* **Per-symbol vs global:** A negative `record_limit` returns up to N records per symbol; a positive value returns N records total, distributed by recency.
* **Compatibility:** N-Query works with both single-dataset and superquery queries. When both `as_of` and `back_to` are set, `record_limit` is ignored and a standard time-range query is executed.
## Superquery parameters
A **superquery** is triggered when `name` is a **list** of dataset names. It merges and resamples data from multiple datasets into a single result.
| Parameter | Default | Description |
| ---------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `superq_resample_rule` | `"1D"` | Resampling interval. Accepts pandas-style strings (e.g. `"5T"`, `"1D"`) or Liberator-native values: fixed (`1s`–`60s`, `1m`–`240m`, `1d`, `7d`) or calendar (`day`, `wk`, `mo`, `qtr`, `yr`). |
| `superq_fill_forward` | `True` | Fill forward missing values. Only `false` is currently supported when using aggregates. |
| `superq_aggregates` | `False` | Aggregation spec. `False` / `None` disables aggregation; a dict provides per-dataset record-level (`last_timestamp`, `first_timestamp`) and column-level aggregates (see below). |
### Column-level aggregate aliases
| Alias | Meaning | Description |
| ----- | ------- | --------------------------- |
| `o` | first | First value in the bucket |
| `h` | high | Maximum value in the bucket |
| `l` | low | Minimum value in the bucket |
| `c` | last | Last value in the bucket |
| `s` | sum | Sum of values in the bucket |
Record-level aggregates (`last_timestamp`, `first_timestamp`) apply to the entire dataset per bucket. Output columns are named `{dataset}_{column}_{alias}` (e.g. `trades_price_o`, `trades_volume_s`) for column-level aggregates, and `{dataset}__first_timestamp` / `{dataset}__last_timestamp` for record-level aggregates.
**Dict format example:**
```json theme={null}
{
"my_dataset": [
"last_timestamp",
"first_timestamp",
{
"price": ["o", "h", "l", "c"],
"volume": ["s"],
"datetime_utc": []
}
]
}
```
## Authentication & authorization
| Parameter | Default | Description |
| --------- | ------- | --------------------------------------------- |
| `user` | `None` | Authorized user name (used for entitlements). |
| `system` | `None` | System identifier. Typically `"API"`. |
| `token` | `None` | Authentication token. |
The `user` and `token` parameters can be provided per-query or configured once at the SDK level:
* **Python:** Set via the `liberator.json` file in your working directory.
* **JavaScript:** Use `liberator.set_default("query", credentials)` to avoid passing them every time.
* **RESTful:** Include in each request's JSON body.
* **Other SDKs:** Loaded from `liberator.json` in the working directory.
## Connection & transfer
| Parameter | Default | Description |
| ------------ | ------- | ---------------------------------------------- |
| `compress` | `False` | Compress the response on the wire. |
| `json_xfer` | `False` | Use JSON transfer format instead of Arrow IPC. |
| `batch_size` | `25000` | Rows per batch for chunked / streaming output. |
## Cache & derived data
| Parameter | Default | Description |
| -------------------------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `force_regen` | `False` | Bypass caches (e.g. snapfresh, superquery cache) and regenerate results. |
| `dependent_partition_name` | `None` | Partition UUID for dependency tracking (used for cache invalidation). |
| `skip_validation` | `None` | Tri-state override for cache\_manager validation. When unset, the server default from environment is used. |
| `skip_discovery` | `None` | Tri-state override for partition discovery. When unset, the server default from environment is used. |
## Streaming & debug
| Parameter | Default | Description |
| ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `force_streaming` | `False` | Enable incremental streaming for lower time-to-first-byte. Supported only for single-table, non–N-Query reads. |
| `debug_stream` | `False` | Enable debug / progress output. Language-specific type (e.g. `stderr()` in R, `std::ostream*` in C++, `System.IO.TextWriter` in C#). |
| `warning_stream` | `None` | Output stream for warning information. Same type conventions as `debug_stream`. |
## SDK connection options
These options configure the SDK client rather than the query itself.
| Parameter | Default | Description |
| --------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `url` | From Profile download | CloudQuant Data Liberator server URL. Preconfigured in clients downloaded from your Liberator Profile page. Override for a specific IP or port, e.g. `http://127.0.0.1:47753`. |
## Example queries
### Basic time-range query
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL", "MSFT"],
back_to="2025-01-01 00:00:00",
as_of="2025-01-02 00:00:00",
fields=["Price", "Volume"],
order="asc_strict"
)
```
### Symbol pattern filter (2.4+)
```python theme={null}
result = liberator.query(
name="energy_contracts",
symbols="PWR%", # LIKE wildcard; mix with exact keys in a list if needed
back_to="2026-08-01",
as_of="2026-08-02",
)
```
### LKV (last-known-value) query
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
max_lookback=3 # last 3 partitions
)
```
### N-query: most recent 200 records (backward from `as_of`)
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
record_limit=200,
as_of="2025-01-15 00:00:00"
)
```
### N-query: first 200 records (forward from `back_to`)
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
record_limit=200,
back_to="2025-01-01 00:00:00"
)
```
### N-query per-symbol (negative limit)
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL", "MSFT"],
record_limit=-10, # up to 10 records per symbol
as_of="2025-01-15 00:00:00"
)
```
### Superquery with aggregates
```python theme={null}
result = liberator.query(
name=["dataset1", "dataset2"],
symbols=["AAPL"],
back_to="2021-01-01",
as_of="2021-03-01",
superq_resample_rule="5T",
superq_fill_forward=False,
superq_aggregates={
"dataset1": [
"last_timestamp",
"first_timestamp",
{
"Trade Price": ["o", "h", "l", "c"],
"volume": ["s"]
}
]
}
)
```
### Raw SQL
```python theme={null}
result = liberator.query(
sql='SELECT * FROM "my_dataset" WHERE muts >= 1704067200000000 AND muts < 1704153600000000 AND symbol = $$AAPL$$',
user="my_user",
system="API"
)
```
### Parameter-based query with `where`
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2025-01-01 00:00:00",
as_of="2025-01-02 00:00:00",
where="volume > 0",
fields=["Price", "Volume"],
order="asc_strict"
)
```
# C++ SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/cpp/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator C++ API.
# C++ SDK getting started
The CloudQuant Data Liberator C++ SDK uses Apache Arrow for high-performance data streaming with a generator-based iteration pattern.
## Required security files
Place the following files from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Type reference
The SDK uses `std::variant` types for flexible argument and return handling:
```cpp theme={null}
using Arg = std::variant,
std::ostream*>;
using Func = std::function,
std::shared_ptr>()>;
using Result = std::variant, Func>;
```
## Quick start
### Query with get\_table
The simplest approach converts results directly into an Arrow Table:
```cpp theme={null}
Liberator liberator;
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
}));
(void)arrow::PrettyPrint(*table, arrow::PrettyPrintOptions(0, 1), &std::cout);
```
### Query with generator pattern
For streaming or large results, use the generator pattern to iterate through record batches:
```cpp theme={null}
Liberator liberator;
Liberator::Result ptr;
ptr = liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s}
});
auto generator = *std::get_if(&ptr);
for (auto res = generator(); res.index(); res = generator())
{
auto batch = *std::get_if<2>(&res);
(void)arrow::PrettyPrint(*batch, arrow::PrettyPrintOptions(0, 1),
&std::cout);
}
```
# C# SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/csharp/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator C# API.
# C# SDK getting started
The CloudQuant Data Liberator C# SDK streams data as Arrow Record Batches, which can be converted to Microsoft DataFrames.
## Required security files
You need the following files from your Downloads ZIP, placed in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
`liberator.json` must be in your working directory.
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Quick start
### Query and iterate record batches
```csharp theme={null}
Liberator liberator;
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DateTime.Now.ToString("hh:mm:ss.fff") + " " + DataFrameUtils.PrettyPrint(df));
}
```
### Get DataFrame directly
```csharp theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
var df = liberator.get_dataframe(res);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
```
# Configuring Your Account
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/configuring-account
How to set up your CloudQuant account credentials in the Excel Plug-In.
# Excel plug-in: configuring your account
Before accessing any data, you need account information from CloudQuant: **User ID**, **Token**, and **Base URL**.
## Getting started
Download the Excel Plug-In from your Liberator Profile page.
In Liberator **2.4** and later, the CloudQuant Excel plugin is also available from the Liberator client UI — you do not need a separately shared installer.
After installation, Excel displays a new **CloudQuant** Ribbon menu item.
## Configuration steps
1. Select the **CloudQuant** tab
2. Select the **Account** icon to open settings
3. Enter your account credentials in the dialog
## Required information
* **User ID**, **Token**, and **Base URL** — provided on your Liberator Profile page (no trailing slash on Base URL)
When you select OK, the application verifies your login credentials. Once verified, the remaining ribbon buttons are enabled so you can create, run, or interact with queries.
# Creating a Query Using the Ribbon Bar Wizard
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/ribbon-bar-wizard
Step-by-step guide to creating queries in the Excel Plug-In using the Ribbon Bar Wizard.
# Excel plug-in: creating a query using the Ribbon Bar wizard
The **Create** icon opens a Wizard that guides you through the new query process.
If the Wizard ribbon is grayed out, open the right side task pane and select "Query". The Wizard will now be active.
## Step-by-step process
### 1. Provide a query name
You can assign a name to your query at the beginning of the dialog. The system provides a default query name if you skip this step, though naming your query is considered best practice.
### 2. Select dataset from the tree
Available datasets are organized hierarchically by group. Double-click the first arrow or select the second arrow to explore group contents.
#### Searching for datasets by name
A search text box lets you locate specific datasets by name rather than browsing the tree structure.
### 3. Select symbols
Symbols are central to most queries. You have three options:
* Type in a symbol manually
* Link symbols to a spreadsheet location
* Retrieve all available symbols
If you do not specify a set of (one or more) symbols, you will receive all the symbols in the dataset.
### 4. Time range or point in time selection
This dialog allows specification of temporal parameters for the query.
Some datasets contain large amounts of data. Excel has limitations on how many rows of data can appear in a spreadsheet tab. It is best practice to query for a specific symbol first.
### 5. Select data location
Select where results should appear. The **Active Cell** option places data at your current spreadsheet location.
Adjust the active cell location by selecting a cell in the spreadsheet before selecting Finish.
### 6. View results
Query results appear as a formatted table within the spreadsheet.
# Creating a Query Using the Pop Out Task Pane
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/task-pane
How to use the Excel Plug-In task pane to create, manage, and run queries.
# Excel plug-in: creating a query using the pop out Task Pane
## The pop out Task Pane
A pop-out task pane appears on the right side of your spreadsheet and can be minimized, opened, or closed.
If the pane is not visible, use the "Restore Task Pane" button in the CloudQuant menu.
## Pane features
### Query tab
The Query tab displays queries you have created, allowing you to edit, re-run, or delete them. Queries are stored in the spreadsheet where they are created, enabling sharing with other licensed users.
The bottom section shows query results. If a query fails, it displays the reason.
### Datasets tab
The Datasets tab lets you create queries directly from the task pane as a shortcut to using the Wizard:
1. **Step 1**: Select your dataset
2. **Step 2**: Name your query, enter symbols, and select your time range
3. **Step 3**: Select data placement — "New Sheet", "Active Cell", or "Replace Table" (for editing existing queries)
When the DATASET pane is selected, ribbon bar options are grayed out. Select the QUERY tab to restore them.
### Resizing the Task Pane
Drag the left frame edge until your cursor becomes a double arrow, then click and drag to resize. Columns can also be resized using double-arrows on column dividers.
### Restoring the pane
Use the restore icon in the CloudQuant ribbon menu if you close the pane entirely.
## Result sets
Results are placed into tables named after your query. If data goes to a new sheet, that sheet is also named after the query.
Give queries specific, simple names. Access named ranges through Excel's Name Manager (accessible via the lightbulb icon).
# Understanding Queries
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/understanding-queries
Overview of point-in-time and time series query types in the Excel Plug-In.
# Excel plug-in: understanding queries
Queries with CloudQuant Data Liberator allow you to retrieve two distinct types of data: **point-in-time data** or **time series data**.
## Point-in-time data
When requesting point-in-time data, you typically query information for the current moment. However, you can specify any particular point in time to examine how data appeared at that previous moment — known as the "as of" date in CloudQuant Data Liberator terminology.
When reading a dataset with point-in-time queries, the system returns data for your specified time. This lets you retrieve historical snapshots showing what information was known at earlier dates.
Think of point-in-time data structurally: one row for each given trading symbol.
## Time series data
Time series data returns multiple rows for each trading symbol from your specified dataset. For instance, requesting Minute Bar stock prices over a ten-minute period generates 10 data rows per symbol.
# Java SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/java/getting-started
Prerequisites and setup guide for using the Java CloudQuant Data Liberator External API.
# Java SDK getting started
The CloudQuant Data Liberator Java SDK uses Apache Arrow for efficient data streaming and provides a generator-based iteration pattern.
## Prerequisites
```bash Debian/Ubuntu theme={null}
sudo apt-get install ca-certificates-java openjdk-17-jdk openjdk-17-jre maven
```
```bash RHEL/CentOS theme={null}
sudo dnf install ca-certificates java-17-openjdk java-17-openjdk-devel maven
```
Run the `./build_maven.sh` script to build the Maven project and install the project dependencies.
## Required security files
Place the following files from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
`liberator.json` must be in your working directory.
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Quick start
```java theme={null}
Liberator liberator = new Liberator();
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("symbols", "AAPL");
}});
for (Liberator.QueryResult res : liberator.Generate()) {
Object res_id = res.GetId();
Object data = res.GetData();
if (data instanceof org.apache.arrow.vector.VectorSchemaRoot) {
org.apache.arrow.vector.VectorSchemaRoot recordBatch =
(org.apache.arrow.vector.VectorSchemaRoot) data;
printRecordBatch(recordBatch);
} else if (data instanceof javax.json.JsonValue) {
javax.json.JsonValue jsonValue = (javax.json.JsonValue) data;
System.out.println(jsonValue);
}
}
```
# JavaScript SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/javascript/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator NodeJS module.
# JavaScript SDK getting started
The CloudQuant Data Liberator NodeJS module provides promise-based access to CloudQuant datasets.
## Installation
Install the `liberator_module` package in your NodeJS project.
## Required security files
Place the following file from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
## Quick start
```javascript theme={null}
const liberator = require('liberator_module').liberator;
const fs = require('fs');
let credentials = {
user: JSON.parse(fs.readFileSync('liberator.json')).user,
token: JSON.parse(fs.readFileSync('liberator.json')).token,
};
liberator.set_default("query", credentials);
let params = {
as_of: "2021-01-12 23:59:59",
back_to: "2021-01-12 00:00:00",
symbols: "AAPL",
name: "daily_bars",
};
liberator.query(params).then(function(results) {
console.log(results);
});
```
Different datasets have varying update frequencies. Some datasets (like earnings reports) update quarterly, while others update intraday. Time series queries may return limited results depending on data availability.
# Advanced - Query Data Using requests.post RESTful API
Source: https://knowledge.cloudquant.com/api-reference/python/advanced-rest-api
How to query CloudQuant Data Liberator data directly using Python's requests.post method with the RESTful API.
# Advanced - query data using requests.post RESTful API
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `cert=cert` argument and `pfx_to_pem` helper shown below are only needed for releases prior to 2.0.
## requests.post from Python to access CloudQuant Data Liberator data
`requests.post` is an HTTP method you can use to send query arguments to the CloudQuant Data Liberator server for processing.
## Post arguments
* **`{base_url}/liberator/query`** — The query endpoint. Use the base URL from your Liberator Profile page download.
* **`data=json.dumps(...)`** - This `json.dumps` section provides the body of the query with these parameters:
* `compress`
* `json_xfer`
* `user`
* `token`
* `name`
* `as_of`
* `back_to`
* `symbols`
* `system`
* **`headers={'Content-Type':'application/json'}`**
* **`cert=cert`** - cert comes from `with pfx_to_pem('liberator.pfx','') as cert:`
## Query the last known value(s)
To get the last known value in any dataset, do not provide the `as_of` or `back_to` arguments. This will give you the last known value for the given dataset and symbols.
## Query for a time series result
Adding the `back_to` argument for any query will give you the time series data all the way back to the specified date.
Every dataset has different data frequencies. Use short time frames until you become familiar with the datasets. Some datasets are quite large, and an over-extended request can return an extremely large amount of data.
## json.dumps components
| Argument | Description | Type | Example |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ------------------------------------ |
| `symbols` | The security trading symbol(s) you wish to query | String, or List | `"symbols":["AAPL","TLT","GOOG"]` |
| `name` | The name of the dataset **(Required)** | String | `"name": "daily_bars"` |
| `as_of` | This value can be any past date so that you can see the data as it was known on the "as of" date. Defaults to now. | String `YYYY-MM-DD HH:MM:SS` (HH:MM:SS optional) | `"as_of":"2020-11-22 19:51:31"` |
| `back_to` | The date where the return dataset should begin. | String `YYYY-MM-DD HH:MM:SS` (HH:MM:SS optional) | `"back_to":"2020-01-01"` |
| `url` | Optional. The URL of the CloudQuant Data Liberator server. Preconfigured in clients downloaded from your Liberator Profile page. | String | `'http://127.0.0.1:47753'` |
| `system` | The name of the authorized system from which you are querying. | String | `"system":"API"` |
| `compress` | The data compression method on the wire. | Boolean | `True` or `False` |
| `json_xfer` | JSON transfer. Usually False. | Boolean (Always False) | `"json_xfer":False` |
| `user` | The user identifier (as assigned by CloudQuant) | String | `"user":"myUserID"` |
| `token` | The user's assigned token | String | `"token":"mypersonal-private-token"` |
## Example: calling the CloudQuant Data Liberator REST API with Python
```python theme={null}
import sys
import zlib
import json
import base64
import requests
import tempfile
import contextlib
import pyarrow as pa
import OpenSSL.crypto
@contextlib.contextmanager
def pfx_to_pem(pfx_path, pfx_password):
'''Decrypts the .pfx file to be used with requests.'''
with tempfile.NamedTemporaryFile(suffix='.pem') as t_pem:
t_pem.close()
f_pem = open(t_pem.name, 'wb')
pfx = open(pfx_path, 'rb').read()
p12 = OpenSSL.crypto.load_pkcs12(pfx, pfx_password)
f_pem.write(OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM, p12.get_privatekey()))
f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, p12.get_certificate()))
ca = p12.get_ca_certificates()
if ca is not None:
for cert in ca:
f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert))
f_pem.close()
yield t_pem.name
base_url = '' # from Liberator Profile page download
compressed_transfer = True
# POST the query and prepare for a stream of single-line JSON replies
with pfx_to_pem('liberator.pfx', '') as cert:
r = requests.post(f'{base_url}/liberator/query',
data=json.dumps({"compress": compressed_transfer, "json_xfer": False,
"user": user, "token": token,
"name": name, "as_of": as_of, "back_to": back_to, "symbols": symbols,
"system": "API"}),
headers={'Content-Type': 'application/json'},
cert=cert,
stream=True)
# Ensure that the request was successful
if r.status_code != 200:
print(r.json())
else:
batches = []
for line in r.iter_lines(4096):
# Show progress
print('.', end='')
sys.stdout.flush()
# The response stream contains informational messages other than "batch"
if line[0:14] == b'{"exception":"':
print('Error: ' + json.loads(''.join([chr(c) for c in line]), encoding='latin-1')['value'])
break
if line[0:10] != b'{"batch":"':
continue
# Cut out the payload from the JSON line without json module overhead
decoded = base64.b64decode(line[10:-3])
# Interpret the payload as an Apache Arrow IPC stream
reader = pa.ipc.open_stream(zlib.decompress(decoded) if compressed_transfer else decoded)
# Accumulate RecordBatch objects in a list
batches.extend([batch for batch in reader])
print('')
if batches:
# Create an Arrow Table view on the batches and convert to a pandas DataFrame
df = pa.Table.from_batches(batches).to_pandas()
if 'muts' in df and '_seq' in df:
# By default, the data is sorted by symbol... re-sort by time
df = df.sort_values(['muts', '_seq'])
df.reset_index(drop=True, inplace=True)
print(df)
```
# CloudQuant Charting
Source: https://knowledge.cloudquant.com/api-reference/python/charting
Python library for creating branded financial visualizations including candlestick charts, histograms, line charts, bar charts, and more.
# CloudQuant charting
CloudQuant Charting is a Python library for creating branded financial visualizations using the CloudQuant Data Liberator API.
**Import statement:**
```python theme={null}
from liberator import charting
```
## Chart types
### Histogram
Generates distribution/histogram charts with CloudQuant branding.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame containing data
* `col` - string name of column to visualize
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - support HTML tags like ``, ``, ` `
* `histnorm` - normalization type: `'percent'`, `'probability'`, `'density'`, or `'probability density'`
* `width`, `height` - dimensions in pixels (min 400px)
* `overlay_df`, `overlay_col` - add comparison series
**Example:**
```python theme={null}
import liberator
from liberator import charting
res = liberator.query(name="daily_bars")
df_bars = liberator.get_dataframe(res)
df_bars = df_bars.sort_values(by=['volume'], ascending=[False])
change = df_bars.close - df_bars.open
df_bars["change"] = change
fig = charting.Histogram(
df=df_bars[0:100],
col="change",
title="Change Px Distribution of last trading day's top 100 High Volume Stocks",
width=800,
height=400,
histnorm="probability"
)
fig.show()
```
### Candlestick
Generates OHLC candlestick charts with optional technical studies and overlays.
**Returns:** Single figure or array of figures (depending on studies)
**Required arguments:**
* `df` - DataFrame with OHLC data (minimum 20 bars)
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - chart labels with HTML support
* `width`, `height` - dimensions (min 700px width, 400px height)
* `open`, `high`, `low`, `close`, `timestamp` - column names
* `entry_px`, `entry_time`, `close_px`, `close_time`, `entry_side` - trade annotations
* `overlay_df`, `overlay_col`, `overlay_col2` - secondary axis data
**Supported Studies (Single Figure):**
`BBANDS`, `DEMA`, `EMA`, `HT_TRENDLINE`, `KAMA`, `MA`, `MAMA`, `MIDPOINT`, `MIDPRICE`, `SAR`, `SAREXT`, `SMA`, `T3`, `TEMA`, `TRIMA`, `WMA`
**Supported Studies (Array of Figures):**
`ADX`, `ADXR`, `APO`, `AROON`, `AROONOSC`, `BOP`, `CCI`, `CMO`, `DX`, `MACD`, `MACDEXT`, `MFI`, `MINUS_DI`, `MINUS_DM`, `MOM`, `PLUS_DI`, `PLUS_DM`, `PPO`, `ROC`, `ROCP`, `ROCR`, `ROCR100`, `RSI`, `STOCH`, `STOCHF`, `STOCHRSI`, `TRIX`, `ULTOSC`, `WILLR`, `ATR`, `MACDFIX`, `NATR`, `TRANGE`
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
from datetime import datetime, timedelta
back_to = (datetime.now() + timedelta(days=-360)).strftime('%Y-%m-%d 23:59:59')
symbol = "SPY"
res = liberator.query(name="daily_bars", symbols=symbol, back_to=back_to)
df_bars = liberator.get_dataframe(res)
res = liberator.query(name="eia_gas_prices", back_to=back_to)
gaspx = liberator.get_dataframe(res)
fig = charting.Candlestick(
df=df_bars,
overlay_df=gaspx,
overlay_col="retail_gas_price",
height=700,
width=800,
xlabel="Date",
title="SPY vs Retail Gas Prices"
)
fig.show()
```
### LineChart
Creates line charts with support for studies and secondary Y-axis.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `cols` - list of column names to plot
* `x_column` - column name for X-axis (defaults to index)
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - chart labels with HTML support
* `width`, `height` - dimensions
* `y2axis_name` - activates secondary Y-axis
* `overlay_df`, `overlay_cols` - secondary axis data
* `study` - technical study name
* `study_columns` - columns for study calculation
* `timeperiod` - study periods
**Supported Studies:**
`BBANDS`, `MAMA`, `DEMA`, `EMA`, `HT_TRENDLINE`, `KAMA`, `MA`, `MIDPOINT`, `MIDPRICE`, `SAR`, `SAREXT`, `SMA`, `T3`, `TEMA`, `TRIMA`, `WMA`
**Example:**
```python theme={null}
import cloudquantcharting
import liberator
res = liberator.query(
symbols='AMZN',
name='Twitter',
back_to='12/22/2020 08:00:00',
as_of='12/22/2020 16:00:00'
)
twitter_df = liberator.get_dataframe(res)
fig = cloudquantcharting.LineChart(
df=twitter_df,
cols=['s-score', 's-buzz', 's', 's-mean', 'sv-score', 's-dispersion', 's-buzz', 's-delta'],
x_column="timestamp"
)
fig.show()
```
### BarChart
Generates bar charts, including stacked bars for repeated X values.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `xcol` - column name for X-axis
* `ycol` - numeric column for Y-axis
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - labels with HTML support
* `width`, `height` - dimensions
* `singlecolor` - boolean for uniform color (defaults `True`)
* `orientation` - `'v'` (vertical) or `'h'` (horizontal)
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
res = liberator.query(name="daily_bars")
df_bars = liberator.get_dataframe(res)
df_bars = df_bars.sort_values(by=['volume'], ascending=[False])
fig = charting.BarChart(
df=df_bars[0:20],
xcol="symbol",
ycol="spread",
title="Spread Price of High Volume Stocks",
width=800,
height=400
)
fig.show()
```
### PieChart
Generates pie/donut charts with customizable hole size.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `labelcol` - column name for labels
* `valuecol` - numeric column for slice size
**Optional arguments:**
* `title` - chart title with HTML support
* `width`, `height` - dimensions
* `hole` - donut hole size (0-1, where 0 = no hole)
* `colors` - color sequence: default CQ palette, or `'reds'`, `'greens'`, `'blues'`
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
values = [
['Jan', '150000', 'USD'],
['Feb', '200000', 'USD'],
['Mar', '330000', 'USD'],
['Apr', '440000', 'USD'],
['May', '555000', 'USD'],
['June', '355000', 'USD'],
['July', '150000', 'USD'],
['Aug', '130000', 'USD'],
['Sept', '330000', 'USD'],
['Oct', '440000', 'USD'],
['Nov', '130000', 'USD'],
['Dec', '330000', 'USD']
]
df = pd.DataFrame(values, columns=["month", "sales", "currency"])
title = "Sales by month"
fig = charting.PieChart(
df=df,
labelcol='month',
valuecol="sales",
title=title,
height=700
)
fig.show()
```
### GroupedBarChart
Generates grouped bar charts comparing multiple numeric columns.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `groups` - column name for grouping
* `values` - list of numeric column names
**Optional arguments:**
* `title`, `group_label`, `value_label` - labels with HTML support
* `width`, `height` - dimensions
* `colors` - color list (cycles through bars)
* `orientation` - `'v'` or `'h'`
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
dataset_name = 'reddit_wallstreetbets_comments'
df = liberator.get_dataframe(liberator.query(name=dataset_name))
df = df.loc[df['vader_body_sentiment_compound'] > 0]
df = df.sort_values(by=['vader_body_sentiment_compound'], ascending=[False])
charting.GroupedBarChart(
df=df[0:5],
groups="symbol",
values=["vader_body_sentiment_pos", "vader_body_sentiment_compound", "textblob_body_sentiment_subjectivity"],
title="WS Bets Sentiment",
group_label="Symbol",
value_label="sentiment score"
)
```
### ScatterPlot
Generates scatter plots with optional bubble sizing and color coding.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `x_column` - numeric column for horizontal axis
* `y_column` - numeric column(s) for vertical axis
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - labels with HTML support
* `width`, `height` - dimensions
* `size_column` - numeric column for bubble size (negative values show red/green)
* `size_multiplier` - multiplier to adjust bubble sizes
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
dataset_name = 'reddit_wallstreetbets_comments'
df = liberator.get_dataframe(liberator.query(name=dataset_name))
df = df.loc[df['vader_body_sentiment_compound'] > 0]
df = df.sort_values(by=['vader_body_sentiment_compound'], ascending=[False])
fig = charting.ScatterPlot(
df=df[0:200],
x_column="textblob_body_sentiment_subjectivity",
y_column=['vader_body_sentiment_neu'],
title="WS Bets Sentiment",
size_multiplier=.15,
width=800,
height=800
)
fig.show()
```
### addNotes
Adds annotations/callouts to existing charts for highlighting specific data points.
**Returns:** `plotly.graph_objs._figure.Figure`
**Arguments:**
* `thefig` - figure from a CloudQuant charting function
* `notes` - list of dictionaries with `x`, `y`, and `note` keys
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
from datetime import datetime, timedelta
back_to = (datetime.now() + timedelta(days=-2)).strftime('%Y-%m-%d 23:59:59')
symbol = "AAPL"
res = liberator.query(name="minute_bars", symbols=symbol, back_to=back_to)
df_bars = liberator.get_dataframe(res)
fig = charting.Candlestick(df=df_bars, title=symbol)
mydata = []
data = {'x': df_bars.timestamp[7], 'y': df_bars.open[7], 'note': "T1"}
mydata.append(data)
data = {'x': df_bars.timestamp[155], 'y': df_bars.open[155], 'note': "T2"}
mydata.append(data)
data = {'x': df_bars.timestamp[1000], 'y': df_bars.open[1000], 'note': "T3"}
mydata.append(data)
fig = charting.addNotes(fig, mydata)
fig.show()
```
# Python SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/python/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator Python API from your own environment.
# Python SDK getting started
The CloudQuant Data Liberator Python SDK provides the simplest way to query and work with CloudQuant datasets from any Python environment.
## Prerequisites
* Python 3
## Installation
Install the third-party dependencies Liberator needs. The `liberator.py` client itself is not on PyPI — download it from your Liberator Profile page (see [Required security files](#required-security-files) below).
```bash Conda theme={null}
conda create -n liberator python=3.7
conda activate liberator
conda install -c anaconda requests pyopenssl cryptography six pytz
conda install pandas
conda install -c conda-forge httpx async_generator pyarrow=8.0.0
```
```bash pip theme={null}
python -m venv liberator
source liberator/bin/activate # Windows: liberator\Scripts\activate
pip install requests pandas httpx async_generator "pyarrow==8.0.0" pyOpenSSL cryptography six pytz
```
## Required security files
You need the following files from your Downloads ZIP file:
* **`liberator.py`** — The CloudQuant Data Liberator API for Python
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
Place `liberator.py` in the same folder as your code or in your Python `site-packages` directory. Place `liberator.json` (and `liberator.pfx` if needed) in your **working directory**, or set explicit paths before querying:
```python theme={null}
import liberator
liberator.auth = '/path/to/liberator.json'
liberator.pfx = '/path/to/liberator.pfx' # releases prior to 2.0 only
```
`liberator.json` is resolved from your working directory unless you set `liberator.auth`.
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
When using Jupyter Notebooks, set the working directory to the location containing `liberator.json` (and `liberator.pfx` if you are on a release prior to 2.0). See [this reference](https://kegui.medium.com/change-jupyter-notebook-startup-folder-windows-608dfcfdc104) for changing the Jupyter Notebook startup folder on Windows.
## Optional: CloudQuant charting
If using CloudQuant Charting (included with the API), install these additional packages:
```bash Conda theme={null}
conda install plotly numpy
conda install -c conda-forge ta-lib
```
```bash pip theme={null}
pip install plotly numpy TA-Lib
```
`TA-Lib` on pip requires the [ta-lib C library](https://github.com/TA-Lib/ta-lib-python) to be installed first. Conda installs both the library and Python bindings together.
## Quick start
```python theme={null}
import liberator
df = liberator.get_dataframe(
liberator.query(
name='daily_bars',
as_of='2020-11-15',
symbols=['AAPL']
)
)
print(df)
```
# R SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/r-language/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator R API.
# R SDK getting started
The CloudQuant Data Liberator R package provides functions for querying datasets and converting results to R DataFrames.
## Required security files
Place the following files from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Quick start
### Query and get DataFrame
```r theme={null}
res = liberator::get_dataframe(liberator::query(
symbols="AAPL",
name="daily_bars",
as_of="2025-01-01",
back_to="2024-12-01"
))
print(res)
```
### Function reference
The `get_dataframe` function accepts a query result and converts it into a DataFrame:
```r theme={null}
get_dataframe <- function(res, muts_order_batches=TRUE)
```
You can send debug output to stderr:
```r theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
symbols="AAPL",
as_of="2019-09-15",
back_to="2019-09-10",
debug_stream=stderr()
)))
```
# RESTful API Getting Started
Source: https://knowledge.cloudquant.com/api-reference/restful/getting-started
Complete guide to querying the CloudQuant Data Liberator RESTful API using curl and standard HTTP tools.
# RESTful API getting started
The CloudQuant Data Liberator API provides RESTful endpoints for querying financial market data. The primary endpoint is `/liberator/query`, which accepts JSON payloads and returns data as JSON objects or Apache Arrow batches.
## Requirements
* `curl`
* `jq`
* `base64`
* Python with `pyarrow` (for Arrow format decoding)
* `pigz` (for compressed Arrow format)
## Authentication
All requests require user credentials and your Liberator base URL (both provided on your Profile page):
```bash theme={null}
URL=
USER=TODO-ADD-YOUR-USERID-HERE
TOKEN=TODO-ADD-YOUR-TOKEN-HERE
```
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `--cert-type P12 --cert liberator.pfx` flags shown in the examples below are only needed for releases prior to 2.0.
## Quick start: JSON queries
### Current value query
```bash theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":["AAPL","IBM"],"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","debug_stream":false}' \
$URL/liberator/query
```
### Time range query
```bash theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","back_to":"2022-12-01","debug_stream":false}' \
$URL/liberator/query
```
### Point-in-time query
```bash theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","as_of":"2022-12-15","debug_stream":false}' \
$URL/liberator/query
```
## Advanced: Apache Arrow format
### Without compression
```bash theme={null}
BATCHES=$(curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":false,"compress":false,"symbols":["AAPL"],"back_to":"2023-01-01","debug_stream":false,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query | jq -r '.[] | select(type=="object") | .batch')
for BATCH in $BATCHES; do
echo $BATCH | base64 -d | python -c "import sys; import pyarrow as pa; print(pa.ipc.open_stream(sys.stdin.buffer.read()).read_next_batch().to_pylist())"
done
```
### With compression
```bash theme={null}
BATCHES=$(curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":false,"compress":true,"symbols":["AAPL"],"back_to":"2023-01-01","debug_stream":false,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query | jq -r '.[] | select(type=="object") | .batch')
for BATCH in $BATCHES; do
echo $BATCH | base64 -d | pigz -dz | python -c "import sys; import pyarrow as pa; print(pa.ipc.open_stream(sys.stdin.buffer.read()).read_next_batch().to_pylist())"
done
```
## Response format
Standard JSON response includes fields such as: `_seq`, `_dsname`, `timestamp`, `symbol`, `open`, `high`, `low`, `close`, `volume`, `vwap`, `spread`, `bidvol`, `askvol`, and `count`.
# Data Catalog
Source: https://knowledge.cloudquant.com/data-catalog/overview
Browse 70+ integrated financial, alternative, and economic datasets available through CloudQuant Data Liberator
# Data catalog - data integrations
CloudQuant Data Liberator integrates with over 70 data providers, offering a comprehensive catalog of financial, alternative, and economic datasets.
## Market & financial data
| Dataset | Description |
| -------------------------------- | ------------------------------- |
| **SpiderRock Options & Futures** | Options and futures market data |
| **CryptoQuote** | Cryptocurrency market data |
| **S\&P Dow Jones Indices** | Index data and analytics |
| **Canari Options** | Options analytics and data |
## Alternative data
| Dataset | Description |
| --------------------------------------- | ------------------------------------ |
| **Alexandria** | Earnings and news sentiment analysis |
| **BMLL Technologies** | Level 3 order book data |
| **Exegy Hidden Order Flow** | Hidden and dark pool order flow |
| **GWI Audience Insight** | Consumer audience analytics |
| **Likefolio Twitter** | Social media sentiment from Twitter |
| **Lunarcrush Crypto Sentiment** | Cryptocurrency social sentiment |
| **Smart Insider (Buybacks)** | Corporate buyback intelligence |
| **New Constructs (Earnings Anomalies)** | Earnings quality analytics |
| **Precision Alpha** | Price prediction signals |
| **Danel SmartScore** | Multi-factor scoring |
## Economic data
| Dataset | Description |
| ------------------------------------------ | ----------------------------- |
| **FRED Release Series** | Federal Reserve Economic Data |
| **Bureau of Economic Analysis GDP** | U.S. GDP data |
| **Bureau of Labor Statistics** | Employment and labor data |
| **U.S. Energy Information Administration** | Energy market data |
| **EuroStat** | European economic statistics |
| **CFTC Commitment of Traders** | Futures positioning data |
## Regulatory & reference data
| Dataset | Description |
| ------------------------------ | --------------------------- |
| **FINRA** | Financial regulatory data |
| **SEC Filing Data** | SEC filings and disclosures |
| **Regalytics Regulatory Data** | Regulatory intelligence |
## Environmental & weather data
| Dataset | Description |
| -------------------- | ----------------------------- |
| **Athenium Weather** | Weather analytics for trading |
| **CustomWeather** | Custom weather data feeds |
| **Ilika Weather** | Weather data and forecasts |
## Additional datasets
| Dataset | Description |
| ---------------------- | ----------------------------------------- |
| **Linkup Jobs** | Job market data |
| **Shipfix Shipping** | Global shipping and freight data |
| **ESG Data Providers** | Environmental, social, governance metrics |
Dataset availability depends on your subscription and entitlements. Use `liberator.datasets()` in Python to see which datasets you have access to. In Liberator **2.4** and later, catalog cards (and `liberator.datasets(schema=True, details=True)`) show date range, symbol count, status, and last updated. Contact CloudQuant to request access to additional datasets.
## Accessing datasets
```python theme={null}
import liberator
# List all datasets you have access to
datasets = liberator.datasets()
print(datasets)
# Query a specific dataset
df = liberator.query(
dataset="your_dataset_name",
symbols=["AAPL"],
as_of="2024-01-01",
back_to="2023-01-01"
)
```
See the [Python Guide](/python-guide/dataset-access) for more details on discovering and querying datasets.
# Aligning Two Datasets Into One
Source: https://knowledge.cloudquant.com/data-science-recipes/aligning-datasets
Strategies for merging two different time series datasets, handling timestamp misalignment, symbol mismatches, and frequency differences.
# Aligning two datasets into one
Merging two different time series datasets into one can be like navigating a minefield — it is one of the trickiest challenges in data science.
## Key challenges
When aligning datasets, you need to consider several critical factors:
1. **Timestamp Availability** — Even if your datasets are timestamped as Daily, you still need to know when the data was available to ensure it can line up.
2. **Symbol Consistency** — Do both datasets contain identical symbols? How do you handle mismatches?
3. **Timeframe Misalignment** — What happens when one dataset operates at 1-minute intervals and another at 5-minute intervals?
4. **Data Expansion Strategy** — When expanding lower-frequency data, should you use first values, last values, or an alternative approach?
5. **Data Aggregation** — When merging in the opposite direction, how do you summarize higher-frequency bars? The choice of average, max, min, or other metrics depends on column content.
## Recommended solution: reindex()
The pandas `reindex()` method allows you to forward fill, back fill, use nearest, or fill with None.
```python theme={null}
df.reindex()
```
For detailed usage, see the [pandas DataFrame.reindex documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reindex.html).
## Alternative: SuperQuery
For datasets within CloudQuant Data Liberator, you can use the **SuperQuery** command to have the system perform the merge automatically. SuperQuery resamples multiple datasets onto a common time axis without manual alignment.
If both of your datasets are available in CloudQuant Data Liberator, SuperQuery is often the simplest approach — it handles the alignment for you. See the [SuperQuery recipe](/data-science-recipes/superquery) for details.
# Extracting Parts of a Date or Time from a Timestamp
Source: https://knowledge.cloudquant.com/data-science-recipes/date-time-extraction
Efficiently extract date and time components from timestamp strings in Python DataFrames using string slicing and pandas methods.
# Extracting parts of a date or time from a timestamp
When working with timestamped market data, you often need to extract specific components like the date, time, or hour. Rather than using slower row-by-row loops, you can use vectorized operations for much better performance.
## String slicing with lambda functions
The primary approach uses `apply()` with lambda functions to extract substrings from timestamp data:
```python theme={null}
df = liberator.get_dataframe(
liberator.query(
name='minute_bars',
symbols='SPY',
as_of='2024-07-26',
back_to='2024-07-25'
)
)
df['Date'] = df.apply(lambda row: str(row.timestamp)[0:10], axis=1)
df['Time'] = df.apply(lambda row: str(row.timestamp)[11:16], axis=1)
df['Hour'] = df.apply(lambda row: str(row.timestamp)[11:13], axis=1)
df['DateTime'] = df.apply(lambda row: str(row.timestamp)[0:16], axis=1)
df['DateHour'] = df.apply(lambda row: str(row.timestamp)[0:13], axis=1)
df[['timestamp', 'Date', 'Time', 'Hour', 'DateTime', 'DateHour']]
```
This creates new columns by slicing the timestamp string at specific positions:
| Column | Slice | Example Output |
| ---------- | --------- | ------------------ |
| `Date` | `[0:10]` | `2024-07-26` |
| `Time` | `[11:16]` | `14:30` |
| `Hour` | `[11:13]` | `14` |
| `DateTime` | `[0:16]` | `2024-07-26 14:30` |
| `DateHour` | `[0:13]` | `2024-07-26 14` |
## Alternative: FLOOR method using Pandas DateTime
You can also use pandas built-in datetime methods:
```python theme={null}
df['seconds'] = df['timestamp'].dt.floor('s')
df['Date'] = pd.to_datetime(df["timestamp"]).dt.date
```
String slicing with lambda functions provides fast processing for extracting timestamp components, making it preferable to manual loops for DataFrame operations.
# Downloading Very Large Datasets
Source: https://knowledge.cloudquant.com/data-science-recipes/large-datasets
Chunk large time series queries into smaller segments to handle network instability and avoid timeouts when downloading market data.
# Downloading very large datasets
For large time series datasets, if you experience network instability, chunk large queries into smaller time segments rather than attempting single monolithic downloads.
## Core strategy
The recommended approach divides large queries into smaller, manageable time-based chunks (typically 30-minute segments). This methodology:
* Reduces network timeout and data corruption likelihood
* Automatically retries failed chunks without losing progress
* Continues downloading even when individual chunks fail
* Provides clear feedback on download progress
## Complete Python implementation
```python theme={null}
import liberator
import pandas as pd
from datetime import datetime, timedelta
from typing import Union, List, Optional
import time
def getdata(dataset: str,
start_date: Union[str, datetime],
end_date: Union[str, datetime],
symbols: Union[str, List[str]],
chunk_minutes: int = 30,
max_retries: int = 3,
retry_delay: float = 1.0,
verbose: bool = True) -> pd.DataFrame:
"""
Download data by breaking the time range into smaller chunks.
Args:
dataset: Dataset name (e.g., 'spiderrock_printsets_indexed_single_tenant')
start_date: Start date/time as string 'YYYY-MM-DD [HH:MM:SS]' or datetime
end_date: End date/time as string 'YYYY-MM-DD [HH:MM:SS]' or datetime
symbols: Single symbol string or list of symbols
chunk_minutes: Minutes per chunk (default 30)
max_retries: Maximum retries per chunk (default 3)
retry_delay: Delay between retries in seconds (default 1.0)
verbose: Print progress messages (default True)
Returns:
pandas.DataFrame: Combined data for the entire time range
"""
# Parse and validate inputs
start_dt = _parse_datetime(start_date)
end_dt = _parse_datetime(end_date)
if isinstance(symbols, str):
symbols = [symbols]
if verbose:
print(f"Downloading {dataset} from {start_dt} to {end_dt} for {len(symbols)} symbols")
print(f"Using {chunk_minutes}-minute chunks")
# Generate time chunks
chunks = _generate_time_chunks(start_dt, end_dt, chunk_minutes)
if verbose:
print(f"Processing {len(chunks)} time chunks...")
# Download each chunk
all_dfs = []
failed_chunks = []
empty_chunks = []
for i, (chunk_start, chunk_end) in enumerate(chunks):
if verbose:
end_display = chunk_end.strftime('%H:%M:%S') if chunk_end.second != 0 else chunk_end.strftime('%H:%M')
print(f" Processing chunk {i+1}/{len(chunks)} ({chunk_start.strftime('%H:%M')} - {end_display})", end="")
# Download chunk with retries
result = _download_chunk_with_retry(
dataset=dataset,
start_time=chunk_start,
end_time=chunk_end,
symbols=symbols,
max_retries=max_retries,
retry_delay=retry_delay,
verbose=verbose
)
if result is None:
# Actual failure (exception occurred)
failed_chunks.append((chunk_start, chunk_end))
if verbose:
print(" - FAILED")
elif len(result) == 0:
# Empty result (no data for this time period)
empty_chunks.append((chunk_start, chunk_end))
if verbose:
print(" - EMPTY")
else:
# Success with data
all_dfs.append(result)
if verbose:
print(f" - SUCCESS ({len(result):,} rows)")
# Report results
if verbose:
total_rows = sum(len(df) for df in all_dfs)
print(f"Complete: {len(all_dfs)} chunks with data, {len(empty_chunks)} empty chunks, {len(failed_chunks)} failed chunks")
print(f"Total rows downloaded: {total_rows:,}")
if len(failed_chunks) > 0:
print(f"Warning: {len(failed_chunks)} chunks failed due to errors")
# Combine all dataframes
if all_dfs:
combined_df = pd.concat(all_dfs, ignore_index=True)
# Sort by time if columns exist
if 'muts' in combined_df.columns and '_seq' in combined_df.columns:
combined_df = combined_df.sort_values(['muts', '_seq'])
combined_df.reset_index(drop=True, inplace=True)
elif 'timestamp' in combined_df.columns:
combined_df = combined_df.sort_values('timestamp')
combined_df.reset_index(drop=True, inplace=True)
return combined_df
else:
print("No data downloaded successfully")
return pd.DataFrame()
def _parse_datetime(dt_input: Union[str, datetime]) -> datetime:
"""Parse string or datetime input into datetime object."""
if isinstance(dt_input, datetime):
return dt_input
dt_str = str(dt_input).strip()
# Try different datetime formats
formats = [
'%Y-%m-%d %H:%M:%S.%f', # Full datetime with microseconds
'%Y-%m-%d %H:%M:%S', # Full datetime
'%Y-%m-%d %H:%M', # Date with hour:minute
'%Y-%m-%d', # Date only
]
for fmt in formats:
try:
return datetime.strptime(dt_str, fmt)
except ValueError:
continue
raise ValueError(f"Unable to parse datetime: {dt_input}")
def _generate_time_chunks(start_dt: datetime, end_dt: datetime, chunk_minutes: int) -> List[tuple]:
"""Generate list of (start, end) datetime tuples for chunks."""
chunks = []
current_start = start_dt
chunk_delta = timedelta(minutes=chunk_minutes)
while current_start < end_dt:
current_end = min(current_start + chunk_delta, end_dt)
chunks.append((current_start, current_end))
current_start = current_end
return chunks
def _download_chunk_with_retry(dataset: str,
start_time: datetime,
end_time: datetime,
symbols: List[str],
max_retries: int,
retry_delay: float,
verbose: bool = True) -> Optional[pd.DataFrame]:
"""Download a single time chunk with retry logic."""
# Format times as strings for liberator
start_str = start_time.strftime('%Y-%m-%d %H:%M:%S')
end_str = end_time.strftime('%Y-%m-%d %H:%M:%S')
for attempt in range(max_retries):
try:
# Query the data
query_result = liberator.query(
name=dataset,
symbols=symbols,
back_to=start_str,
as_of=end_str
)
# Convert to dataframe
df = liberator.get_dataframe(query_result)
# Return the dataframe (could be empty, but that's not a failure)
return df
except Exception as e:
if attempt < max_retries - 1: # Don't sleep on last attempt
time.sleep(retry_delay)
else:
if verbose:
print(f" Failed chunk {start_time.strftime('%H:%M')}-{end_time.strftime('%H:%M')} after {max_retries} attempts: {str(e)}")
return None # Return None only on actual exception
return None
# Example usage functions
def download_single_day(dataset: str, date: str, symbols: Union[str, List[str]], **kwargs) -> pd.DataFrame:
"""
Convenience function to download a single day's data.
Args:
dataset: Dataset name
date: Date as 'YYYY-MM-DD'
symbols: Symbol(s) to download
**kwargs: Additional arguments passed to getdata()
Returns:
pandas.DataFrame: Day's data
"""
start_date = f"{date} 00:00:00"
end_date = f"{date} 23:59:59"
return getdata(
dataset=dataset,
start_date=start_date,
end_date=end_date,
symbols=symbols,
**kwargs
)
def download_date_range(dataset: str,
start_date: str,
end_date: str,
symbols: Union[str, List[str]],
**kwargs) -> pd.DataFrame:
"""
Download data across multiple days.
Args:
dataset: Dataset name
start_date: Start date as 'YYYY-MM-DD'
end_date: End date as 'YYYY-MM-DD'
symbols: Symbol(s) to download
**kwargs: Additional arguments passed to getdata()
Returns:
pandas.DataFrame: Multi-day data
"""
start_dt = _parse_datetime(f"{start_date} 00:00:00")
end_dt = _parse_datetime(f"{end_date} 23:59:59.999")
return getdata(
dataset=dataset,
start_date=start_dt,
end_date=end_dt,
symbols=symbols,
**kwargs
)
```
## Usage examples
### Single day, multiple symbols
```python theme={null}
df1 = download_single_day(
dataset='spiderrock_printsets_indexed_single_tenant',
date='2025-09-08',
symbols=['AAPL', 'GOOGL', 'MSFT']
)
```
### Custom time range
```python theme={null}
df2 = getdata(
dataset='spiderrock_printsets_indexed_single_tenant',
start_date='2025-09-08 09:30:00',
end_date='2025-09-08 16:00:00',
symbols='AAPL',
chunk_minutes=15,
verbose=False
)
```
### Multi-day download
```python theme={null}
df3 = download_date_range(
dataset='spiderrock_printsets_indexed_single_tenant',
start_date='2025-09-03',
end_date='2025-09-06',
symbols=['AAPL', 'GOOGL'],
chunk_minutes=120 # 2-hour chunks for longer periods
)
```
## Sample output
```
Downloading spiderrock_printsets_indexed_single_tenant from 2025-09-03 00:00:00
to 2025-09-06 23:59:59.999000 for 2 symbols
Using 120-minute chunks
Processing 48 time chunks...
Processing chunk 1/48 (00:00 - 02:00) - EMPTY
Processing chunk 2/48 (02:00 - 04:00) - EMPTY
Processing chunk 3/48 (04:00 - 06:00) - EMPTY
Processing chunk 4/48 (06:00 - 08:00) - EMPTY
Processing chunk 5/48 (08:00 - 10:00) - SUCCESS (143,762 rows)
Processing chunk 6/48 (10:00 - 12:00) - SUCCESS (168,790 rows)
...
Complete: 13 chunks with data, 35 empty chunks, 0 failed chunks
Total rows downloaded: 959,576
```
For multi-day downloads, increase `chunk_minutes` to reduce the number of API calls. A value of 120 (2 hours) works well for longer date ranges.
The chunking approach is designed for network resilience. If you are on a stable connection and downloading moderately sized datasets, a single query may be simpler and faster.
# Using Market Calendar (mcal) to Identify Trading Dates
Source: https://knowledge.cloudquant.com/data-science-recipes/market-calendar
Programmatically determine trading dates using the pandas_market_calendars library for NYSE, CME, CBOE, NASDAQ, and other exchanges.
# Using Market Calendar (mcal) to identify trading dates
When retrieving market data, it is beneficial to programmatically determine which dates represent actual trading dates. The **Market Calendar (mcal)** library enables this functionality.
## US stock market calendars
Market Calendar supports multiple US exchanges including NYSE, CME, CBOE, and NASDAQ.
## Implementation example
```python theme={null}
# Get NYSE Schedule for date range
# Documentation: https://pandas-market-calendars.readthedocs.io/en/latest/usage.html
start_date = '2024-01-01'
end_date = '2024-06-01'
import pandas as pd
import pandas_market_calendars as mcal
nyse = mcal.get_calendar('NYSE')
schedule = nyse.schedule(start_date=start_date, end_date=end_date)
schedule
schedule.market_open.iloc[0] # open time first date
schedule.market_close.iloc[0] # close time first date
schedule.market_open.iloc[-1] # open time last date
schedule.market_close.iloc[-1] # close time last date
schedlist = list(schedule.index)
schedlist
```
Although mcal is generally extremely reliable, it is an external library and has on occasion missed a market holiday. Implement defensive coding practices and verify results independently.
The library demonstrates strong reliability for US market holidays but cannot be vouched for with equal confidence regarding international markets.
## Resources
* [Official pandas\_market\_calendars Documentation](https://pandas-market-calendars.readthedocs.io/en/latest/usage.html)
# Melt and Wide to Long - Unpivoting a Pivot
Source: https://knowledge.cloudquant.com/data-science-recipes/melt-and-unpivot
Convert DataFrames from wide to long format using pandas melt and wide_to_long methods.
# Melt and wide to long - unpivoting a pivot
Pandas provides two methods for converting DataFrames from wide to long format (unpivoting).
## Melt method
Unpivot a DataFrame from wide to long format, optionally leaving identifiers set.
```python theme={null}
df.melt()
```
The `melt` function provides flexibility when restructuring data from a wide format into a long format. You can specify which columns serve as identifiers and which become variable-value pairs.
Use `melt` when you need fine-grained control over which columns become identifiers and which become variable-value pairs.
**Documentation:** [pandas.melt](https://pandas.pydata.org/docs/reference/api/pandas.melt.html)
## Wide to long method
Unpivot a DataFrame from wide to long format. Less flexible but more user-friendly than melt.
```python theme={null}
pd.wide_to_long()
```
The `wide_to_long` function offers a more accessible alternative to melt, trading some flexibility for improved usability when performing standard unpivoting operations.
Use `wide_to_long` for simpler, standard unpivoting operations where ease of use is more important than flexibility.
**Documentation:** [pandas.wide\_to\_long](https://pandas.pydata.org/docs/reference/api/pandas.wide_to_long.html)
# A Merge or Join
Source: https://knowledge.cloudquant.com/data-science-recipes/merge-and-join
Combine two DataFrames by columns using pandas merge, join, and merge_asof operations.
# A merge or join
A merge or join operation expands the number of **columns** in a resulting DataFrame by combining data from two tables based on shared values. This differs from concatenation, which combines **rows**.
## Merge vs. concat
| Operation | What it does |
| -------------- | -------------------------------------------------------------------------------------------------------------------- |
| **CONCAT** | Combines rows from two DataFrames (e.g., two 10-row DataFrames yield 20 rows). Generally requires identical columns. |
| **MERGE/JOIN** | Combines columns by matching rows on a shared column, typically increasing column count. |
## pd.merge()
Merging requires a **left** dataset, a **right** dataset, and a common column specified with the `on` parameter.
### Join types
The `how` parameter controls which rows appear in results:
| Join Type | Behavior |
| --------- | ------------------------------------------------------------------------ |
| `inner` | **(DEFAULT)** Only rows where the merge column exists in both DataFrames |
| `left` | All rows from the left DataFrame; `NaN` where right data is missing |
| `right` | All rows from the right DataFrame; `NaN` where left data is missing |
| `outer` | All rows from both DataFrames; `NaN` where matches do not exist |
Use `indicator=True` to add a column showing the merge source (`'both'`, `'left_only'`, `'right_only'`) — especially useful when debugging outer joins.
### Basic examples
```python theme={null}
# Standard merge on a shared column
res = pd.merge(left, right, on='shared_column')
# Left join
res = pd.merge(left, right, on='shared_column', how='left')
# Merge with different column names
res = pd.merge(left, right, left_on='leftcol2',
right_on='rightcol7', how='left')
# Select specific columns from right before merging
res = pd.merge(left, right[['col1', 'col2', 'col3']],
on='shared_column')
# Check overlap before merging
left['datesym'].isin(right['datesym']).value_counts()
```
## merge\_asof()
`merge_asof` offers additional flexibility for time-based merging:
* Merging on multiple values
* Specifying tolerance thresholds
### Example: matching quotes to trades
```python theme={null}
pd.merge_asof(trades, quotes, on='time', by='ticker')
# With tolerance
pd.merge_asof(trades, quotes, on='time', by='ticker',
tolerance=pd.Timedelta('2ms'))
```
If `merge_asof()` fails, try removing the `tolerance` parameter.
## Reference
* [pandas.merge\_asof documentation](https://pandas.pydata.org/docs/reference/api/pandas.merge_asof.html)
# Summarizing Data with a Pivot Table
Source: https://knowledge.cloudquant.com/data-science-recipes/pivot-tables
Use pandas pivot tables to summarize, reorganize, and explore large market datasets by aggregating data into meaningful categories.
# Summarizing data with a pivot table
A pivot table is a data analysis tool that enables you to summarize, reorganize, and explore large datasets. It aggregates data, rotates perspectives, and filters, sorts, and groups information into meaningful categories.
## Key benefits
* **Simplify Complex Data** — Break down large datasets into manageable summaries
* **Identify Trends** — Reveal patterns not immediately obvious in raw data
* **Flexible Analysis** — Quickly change layout and structure to answer different questions
* **Efficiency** — Automate data analysis for faster, regular reporting
## Pandas pivot table syntax
```python theme={null}
import pandas as pd
pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'], aggfunc='sum')
```
## Practical example: analyzing odd lots in AAPL trades
This example demonstrates analyzing AAPL trade data by:
1. Extracting trades between 9:30 AM and 4 PM
2. Creating an `odd_lots` boolean column (shares \< 100)
3. Extracting hour from timestamp
4. Pivoting to count shares by hour and lot size
5. Calculating percentage of odd lots per hour
```python theme={null}
import liberator
%time df = liberator.get_dataframe(
liberator.query(
name='trades',
symbols='AAPL',
as_of='2024-07-30 15:59:58',
back_to='2024-07-30 09:30:01'
)
)
df['odd_lots'] = df['shares'] < 100
df['Hour'] = df.apply(lambda row: str(row.timestamp)[11:13], axis=1)
pivot_table = df.pivot_table(
values='shares',
index='Hour',
columns='odd_lots',
aggfunc='sum',
fill_value=0
)
pivot_table['% Odd Lots'] = (
pivot_table[True] / (pivot_table[True] + pivot_table[False]) * 100
)
pivot_table
```
## Parameter reference
| Parameter | Purpose |
| ------------ | -------------------------------------------------------- |
| `aggfunc` | Specifies aggregation function (`'sum'`, `'mean'`, etc.) |
| `values` | Target column for aggregation |
| `index` | Column(s) defining row groupings |
| `columns` | Column(s) defining column splits |
| `fill_value` | Default value for missing data |
# Python Data Science Shortcuts and Snippets
Source: https://knowledge.cloudquant.com/data-science-recipes/shortcuts-and-snippets
Practical Python solutions for common data science challenges when working with DataFrames and market data.
# Python data science shortcuts and snippets
The solutions to most challenges in data science are best solved by experimentation and failure. This guide focuses on practical, daily-use solutions for common data science tasks.
Large Language Models like ChatGPT can help format data effectively when time is limited. They can be a valuable companion for quick data transformations.
This collection of shortcuts and snippets covers the most frequently encountered data manipulation tasks. While it does not provide comprehensive training on data science fundamentals or Lambda functions, these tools can rapidly transform large DataFrames.
The recipes in this section cover specific topics in depth. Use the navigation to explore individual techniques like date extraction, pivot tables, merging datasets, and more.
# SuperQuery
Source: https://knowledge.cloudquant.com/data-science-recipes/superquery
Resample pandas DataFrames from CloudQuant Data Liberator into a common time axis by querying multiple datasets simultaneously.
# SuperQuery
SuperQuery enables you to resample pandas DataFrames from CloudQuant Data Liberator into a common time axis by querying multiple datasets simultaneously.
## Initial setup
```python theme={null}
import liberator
symbols = ['TSLA']
as_of = '2024-06-07'
back_to = '2024-06-04'
%time df1 = liberator.get_dataframe(
liberator.query(
symbols=symbols,
name='minute_bars',
as_of=as_of,
back_to=back_to
)
)
liberator.get_dataframe(
liberator.query(
name='daily_bars',
symbols=symbols,
as_of=as_of,
back_to=back_to
)
)
```
This loads two independent datasets: minute bars and daily bars.
## Querying multiple datasets together
SuperQuery lets you specify how multiple datasets are reindexed and merged together using the `superq_resample_rule` parameter.
### Daily (1D) resampling
For a three-day query using daily frequency, the result contains three rows with daily bars and the final minute bar of each day:
```python theme={null}
df = liberator.get_dataframe(
liberator.query(
symbols=symbols,
name=['daily_bars', 'minute_bars'],
as_of=as_of,
back_to=back_to,
superq_periods_per_batch_override=5,
superq_resample_rule='1D'
)
)
df
```
### Hourly (60T) resampling
For a three-day query using 60-minute frequency, the result contains 24 rows per day. Daily bar data is timestamped at 8pm:
```python theme={null}
%time df4 = liberator.get_dataframe(
liberator.query(
symbols=symbols,
name=['daily_bars', 'minute_bars'],
as_of=as_of,
back_to=back_to,
superq_periods_per_batch_override=5,
superq_resample_rule='60T'
)
)
df
```
## The superq\_resample\_rule parameter
The `superq_resample_rule` parameter uses pandas "Offset Aliases" to define the resampling frequency.
Common values:
| Rule | Frequency |
| ----- | ---------- |
| `1T` | 1 minute |
| `5T` | 5 minutes |
| `15T` | 15 minutes |
| `60T` | 60 minutes |
| `1D` | 1 day |
For a complete list of offset aliases, see the [pandas time series offset aliases documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases).
# Azure Blob Storage
Source: https://knowledge.cloudquant.com/datasource-config/azure-blob
Configure Azure Blob Storage datasources
# Azure Blob Storage
Azure Blob Storage datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from Azure Blob containers. CloudQuant Data Liberator mounts the container via FUSE.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest on Azure Blob, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | -------------------------- |
| `connection_type` | string | Must be `"azure"` |
| `account` | string | Azure storage account name |
| `key` | string | Storage account access key |
| `container` | string | Blob container name |
### Optional fields
| Field | Type | Default | Description |
| ------------- | ------ | ------- | ------------------------------------------------------------ |
| `endpoint` | string | | Blob endpoint URL (override for sovereign clouds or Azurite) |
| `prefix` | string | `""` | Blob prefix (virtual directory) within the container |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
| `object_key` | string | | Object key pattern for blob selection |
The `endpoint` field is only needed for non-standard Azure environments such as Azure Government, Azure China, or local development with Azurite. For standard Azure public cloud, CloudQuant Data Liberator derives the endpoint from the account name.
### Example connection
```json theme={null}
{
"name": "azure-market-data",
"connection_type": "azure",
"account": "mystorageaccount",
"key": "base64encodedstorageaccountkey==",
"container": "market-data",
"prefix": "equities/daily/"
}
```
Storage account keys grant full access to the entire storage account. Rotate keys regularly and use environment variables or a secrets manager to inject them at deployment time.
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "azure-eod-prices",
"connection_type": "azure",
"account": "caboretlstorage",
"key": "xYzAbCdEfGhIjKlMnOpQrStUvWxYz0123456789==",
"container": "eod-data",
"prefix": "prices/us-equities/"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-eod",
"connection": "azure-eod-prices",
"data_args": {
"file_pattern": "eod_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "date",
"data_dt_format": "%Y-%m-%d",
"data_dt_timezone": "America/New_York",
"data_key_column": "ticker",
"fname_dt_regex": "eod_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "ticker", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "date", "type": "string", "group": "time", "description": "Trade date" },
{ "name": "open", "type": "double", "group": "value", "description": "Open price" },
{ "name": "high", "type": "double", "group": "value", "description": "High price" },
{ "name": "low", "type": "double", "group": "value", "description": "Low price" },
{ "name": "close", "type": "double", "group": "value", "description": "Close price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Daily volume" },
{ "name": "adj_close", "type": "double", "group": "value", "description": "Adjusted close price" }
]
}
```
## Azure setup requirements
Ensure the following before configuring your Azure Blob connection:
1. **Storage account** exists and is accessible from the CloudQuant Data Liberator host
2. **Container** is created within the storage account
3. **Access key** is available (found under Storage Account > Access Keys in the Azure Portal)
4. **Network access** is permitted — if the storage account uses firewall rules, add the CloudQuant Data Liberator host's IP to the allowed list
For initial testing, use Azure Storage Explorer to verify your files are accessible before configuring the CloudQuant Data Liberator connection. This eliminates credential and network issues early.
See the [Azure Blob Storage integration guide](/integrations/azure-blob-storage) for additional setup details.
# CIFS/SMB
Source: https://knowledge.cloudquant.com/datasource-config/cifs
Configure Windows/Samba network share datasources
# CIFS/SMB
CIFS/SMB datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from Windows file shares or Samba servers. This is commonly used in enterprise environments where data is published to shared network drives.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest from CIFS/SMB shares, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ---------- | ------ | ---------------------------------------------------------------------- |
| `username` | string | Windows/Samba domain username |
| `password` | string | Password for authentication |
| `server` | string | Server hostname or IP address |
| `path` | string | UNC path to the share (e.g., `"sharename"` or `"sharename/subfolder"`) |
The `connection_type` for CIFS/SMB connections is inferred from the presence of the `server` and `path` fields. Unlike other connection types, there is no explicit `connection_type` field required.
### Optional fields
| Field | Type | Default | Description |
| -------- | ------ | ------- | -------------------------------------------- |
| `prefix` | string | `""` | Subdirectory within the share to use as root |
### Example connection
```json theme={null}
{
"name": "cifs-shared-drive",
"username": "DOMAIN\\datauser",
"password": "w1nd0wsP@ss",
"server": "fileserver.corp.example.com",
"path": "MarketData",
"prefix": "daily/equities/"
}
```
Use a dedicated service account with read-only permissions on the share. Avoid using personal credentials, as password changes would break the connection.
### Domain authentication
For Active Directory environments, include the domain in the username:
```json theme={null}
{
"username": "CORP\\svc-liberator"
}
```
Or use the UPN format:
```json theme={null}
{
"username": "svc-liberator@corp.example.com"
}
```
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "cifs-risk-reports",
"username": "CORP\\svc-liberator",
"password": "s3rv1ceAcct!",
"server": "nas01.corp.example.com",
"path": "RiskReports",
"prefix": "daily/"
}
```
### Dataset
```json theme={null}
{
"name": "daily-risk-metrics",
"connection": "cifs-risk-reports",
"data_args": {
"file_pattern": "risk_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "report_date",
"data_dt_format": "%Y-%m-%d",
"data_dt_timezone": "America/New_York",
"data_key_column": "portfolio_id",
"fname_dt_regex": "risk_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "portfolio_id", "type": "string", "group": "key", "description": "Portfolio identifier" },
{ "name": "report_date", "type": "string", "group": "time", "description": "Report date" },
{ "name": "var_95", "type": "double", "group": "value", "description": "95th percentile VaR" },
{ "name": "var_99", "type": "double", "group": "value", "description": "99th percentile VaR" },
{ "name": "cvar", "type": "double", "group": "value", "description": "Conditional VaR" },
{ "name": "beta", "type": "double", "group": "value", "description": "Portfolio beta" },
{ "name": "sharpe", "type": "double", "group": "value", "description": "Sharpe ratio" }
]
}
```
## Network requirements
| Requirement | Detail |
| ------------------ | ------------------------------------------------------------------------ |
| **Outbound port** | TCP port 445 (SMB) to the file server |
| **DNS resolution** | The server hostname must resolve from the CloudQuant Data Liberator host |
| **Authentication** | NTLM or Kerberos (depending on domain configuration) |
| **SMB version** | SMB 2.0 or higher recommended; SMBv1 is deprecated and insecure |
If the CloudQuant Data Liberator host is Linux-based, install `cifs-utils` for CIFS mount support.
```bash Debian/Ubuntu theme={null}
apt install cifs-utils
```
```bash RHEL/CentOS theme={null}
yum install cifs-utils
```
## Troubleshooting
| Issue | Possible Cause | Solution |
| ------------------------------------ | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Mount fails with "Permission denied" | Incorrect credentials or insufficient share permissions | Verify credentials and ensure the service account has read access to the share |
| Mount fails with "Host is down" | Network connectivity or firewall blocking port 445 | Check firewall rules and verify SMB port is reachable |
| Files not found | Incorrect `path` or `prefix` | Verify the UNC path using `smbclient` or Windows Explorer |
| Encoding errors | Files use non-UTF-8 encoding | Set `encoding` in data\_args (common alternatives: `"latin-1"`, `"cp1252"` for Windows-origin files) |
# FTPS
Source: https://knowledge.cloudquant.com/datasource-config/ftps
Configure FTP over TLS/SSL datasources
# FTPS
FTPS (FTP over TLS/SSL) datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from FTP servers secured with TLS encryption. This is commonly used with legacy financial data feeds and vendor systems that require explicit FTPS connections.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest over FTPS, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | --------------------------------- |
| `connection_type` | string | Must be `"ftps"` |
| `host` | string | FTP server hostname or IP address |
| `user` | string | Username for authentication |
### Optional fields
| Field | Type | Default | Description |
| ------------- | ------ | ------- | -------------------------------------------------------------------------------- |
| `port` | int | `21` | FTP control port number |
| `password` | string | | Password for authentication |
| `key` | string | | Client certificate private key (PEM format) for certificate-based authentication |
| `no_verify` | bool | `false` | Skip TLS certificate verification |
| `prefix` | string | `""` | Remote directory path to use as root |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
Setting `no_verify` to `true` disables TLS certificate validation. This should only be used for testing or when connecting to servers with self-signed certificates on trusted internal networks. Never use this in production with external servers.
You must provide either `password` or `key` for authentication. Password authentication is most common for FTPS. Client certificate authentication (`key`) is used when the server requires mutual TLS.
### Example connection
```json theme={null}
{
"name": "ftps-vendor-feed",
"connection_type": "ftps",
"host": "ftp.securefeed.example.com",
"port": 21,
"user": "datafeed",
"password": "v3nd0rP@ss",
"prefix": "/outgoing/daily/"
}
```
### Example connection (self-signed certificate)
```json theme={null}
{
"name": "ftps-internal-data",
"connection_type": "ftps",
"host": "ftp.internal.corp.net",
"port": 990,
"user": "etl-service",
"password": "internalPass123",
"no_verify": true,
"prefix": "/data/exports/"
}
```
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "ftps-reference-data",
"connection_type": "ftps",
"host": "ftps.refdata.example.com",
"port": 21,
"user": "cq-download",
"password": "refDataP@ss",
"prefix": "/reference/securities/"
}
```
### Dataset
```json theme={null}
{
"name": "security-master",
"connection": "ftps-reference-data",
"data_args": {
"file_pattern": "secmaster_*.csv",
"sep_override": "|",
"encoding": "utf-8",
"data_dt_column": "effective_date",
"data_dt_format": "%Y-%m-%d",
"data_dt_timezone": "America/New_York",
"data_key_column": "cusip",
"fname_dt_regex": "secmaster_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "cusip", "type": "string", "group": "key", "description": "CUSIP identifier" },
{ "name": "effective_date", "type": "string", "group": "time", "description": "Effective date" },
{ "name": "ticker", "type": "string", "group": "value", "description": "Ticker symbol" },
{ "name": "company_name", "type": "string", "group": "value", "description": "Company name" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Primary exchange" },
{ "name": "sector", "type": "string", "group": "value", "description": "GICS sector" },
{ "name": "shares_outstanding", "type": "int64", "group": "value", "description": "Shares outstanding" }
]
}
```
## FTPS vs SFTP
| Feature | FTPS | SFTP |
| --------------------- | ------------------------------- | -------------- |
| Protocol | FTP + TLS | SSH subsystem |
| Default port | 21 (explicit) or 990 (implicit) | 22 |
| Firewall friendliness | Requires data port range | Single port |
| Common use | Legacy financial feeds | Modern systems |
| Certificate | TLS/SSL certificates | SSH host keys |
If your data vendor supports both FTPS and SFTP, prefer SFTP. It uses a single port, simplifies firewall configuration, and is generally easier to manage.
## Network requirements
| Requirement | Detail |
| ------------------ | ----------------------------------------------------------------------------------- |
| **Outbound port** | TCP port 21 (control) + passive data port range to the FTP server |
| **Passive mode** | FTP passive mode is required; ensure the passive port range is open |
| **DNS resolution** | The hostname must resolve from the CloudQuant Data Liberator host |
| **TLS support** | Server must support explicit TLS (AUTH TLS) on port 21, or implicit TLS on port 990 |
# HubSpot
Source: https://knowledge.cloudquant.com/datasource-config/hubspot
Connect HubSpot CRM objects as Liberator datasets for cross-object analytics
# HubSpot
HubSpot connections let Liberator read CRM objects (contacts, companies, deals, and related associations) and expose them as standard Liberator datasets. Once onboarded, HubSpot data uses the same entitlements, query API, and AI assistant surfaces as file- and database-backed sources.
HubSpot connectivity is available in Liberator **2.3** and later.
## Prerequisites
* A HubSpot account with API access to the objects you intend to query
* A HubSpot private app token (or equivalent API credential) with read scopes for those objects
* **Admin** or **Super Admin** role in Liberator to create connections and datasets
## Connect HubSpot
Go to **Connections → Add Connection** and select **HubSpot**.
Paste your HubSpot API token into the credential field. Liberator stores the secret with the connection and does not display it again after save.
Click **Test Connection**. On success, Liberator can list available HubSpot objects for dataset onboarding.
Create a dataset against the connection, choose the HubSpot object (or association view) to expose, and complete the usual column / timestamp mapping steps.
## What you can query
HubSpot-backed datasets support the same Liberator query parameters as other sources (`symbols` / key fields, `as_of`, `back_to`, column selection, and so on), subject to the object schema HubSpot returns. Use them when you want CRM facts alongside market or file-backed datasets without a separate extract pipeline.
## Tips
* Scope the HubSpot token to **read-only** access for the objects you need.
* Prefer one Liberator dataset per HubSpot object you query often; keep association-heavy views as separate datasets when schemas differ.
* After rotating a HubSpot token, update the connection credentials and re-test before users query.
* In Liberator **2.4** and later, HubSpot queries wait about 10 seconds and retry when the CRM API returns a rate-limit error, so large initial syncs are less likely to fail mid-run.
## Related
Connection + dataset architecture and shared configuration fields
Full Liberator 2.3 release notes, including the HubSpot connector
Rate-limit retries and other 2.4 reliability improvements
# Local File (CSV/TSV)
Source: https://knowledge.cloudquant.com/datasource-config/local-file
Configure local or mounted file datasources for CloudQuant Data Liberator
# Local file (CSV/TSV)
Local file datasources read CSV, TSV, or other delimited flat files from a directory on the CloudQuant Data Liberator server or a mounted filesystem. This is the simplest file-based connection type and serves as the foundation for understanding all other file-based sources.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest on this connection, including Parquet, Arrow, Excel, XML, HDF5, PSV (2.1+), and PCAP (2.2+).
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | ---------------------------------------------------- |
| `connection_type` | string | Must be `"file"` |
| `behavior` | string | Must be `"file"` |
| `location` | string | Absolute path to the directory containing data files |
The `location` field should point to a **directory**, not an individual file. CloudQuant Data Liberator will scan the directory for files matching the `file_pattern` in `data_args`.
### Example connection
```json theme={null}
{
"name": "local-trades-connection",
"connection_type": "file",
"behavior": "file",
"location": "/data/trades"
}
```
## Dataset configuration (data\_args)
All file-based datasources share the same `data_args` fields. These control how CloudQuant Data Liberator finds, parses, and interprets your files.
### Required fields
| Field | Type | Description |
| ----------------- | -------------- | ---------------------------------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern to match files, e.g., `"*.csv"`, `"prefix_*.tsv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format string, or special values: `"muts"`, `"uts"`, `"nuts"`, `"datetime"`, `"date"` |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------------------------------------------------- |
| `sep_override` | string | `","` | Delimiter character: `","` (comma), `"\t"` (tab), `"\|"` (pipe), `";"` (semicolon) |
| `encoding` | string | `"utf-8"` | File encoding (e.g., `"utf-8"`, `"latin-1"`, `"ascii"`) |
| `data_dt_timezone` | string | `"UTC"` | Timezone of source data, e.g., `"UTC"`, `"America/New_York"` |
| `fname_dt_regex` | string | | Regex to extract a date from the filename |
| `fname_dt_format` | string | | strptime format for the date extracted by `fname_dt_regex` |
| `fname_dt_timezone` | string | | Timezone of the filename-derived date |
| `fname_dt_nudge` | int | `0` | Microsecond offset applied to filename-derived dates |
| `fname_dt_approx_seconds` | int | | Approximate number of seconds of data per file (used for query optimization) |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order for the resulting Arrow table |
| `arrow_timestamp` | bool | `true` | Whether to generate the human-readable `timestamp` column |
Set `fname_dt_approx_seconds` to `86400` for daily files. This helps CloudQuant Data Liberator skip files outside the query's time range, significantly improving performance for large directories.
## Complete example
Below is a full configuration showing both the connection and a dataset for daily trade CSV files.
### Connection
```json theme={null}
{
"name": "local-daily-trades",
"connection_type": "file",
"behavior": "file",
"location": "/data/daily-trades"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "local-daily-trades",
"data_args": {
"file_pattern": "trades_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "trade_time",
"data_dt_format": "%Y-%m-%d %H:%M:%S",
"data_dt_timezone": "America/New_York",
"data_key_column": "symbol",
"fname_dt_regex": "trades_(\\d{4}-\\d{2}-\\d{2})\\.csv",
"fname_dt_format": "%Y-%m-%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" }
]
}
```
Ensure the CloudQuant Data Liberator process has read permissions on the `location` directory and all files within it. Permission errors will cause silent failures during query execution.
## Tab-separated files (TSV)
For TSV files, set `sep_override` to `"\t"`:
```json theme={null}
{
"data_args": {
"file_pattern": "*.tsv",
"sep_override": "\t",
"data_dt_column": "date",
"data_dt_format": "%Y%m%d",
"data_dt_timezone": "UTC",
"data_key_column": "ticker"
}
}
```
## Composite key example
When the symbol is constructed from multiple columns:
```json theme={null}
{
"data_key_column": [
{ "type": "column", "value": "exchange" },
{ "type": "literal", "value": "_" },
{ "type": "column", "value": "ticker" }
]
}
```
This produces keys like `NYSE_AAPL`, `NASDAQ_MSFT`, etc.
## Multiple datetime columns
When the date and time are in separate columns:
```json theme={null}
{
"data_dt_column": ["trade_date", "trade_time"],
"data_dt_format": ["%Y-%m-%d", "%H:%M:%S.%f"]
}
```
CloudQuant Data Liberator concatenates the columns with a space before parsing, so the effective format becomes `"%Y-%m-%d %H:%M:%S.%f"`.
# SQL Server (MSSQL)
Source: https://knowledge.cloudquant.com/datasource-config/mssql
Configure Microsoft SQL Server datasources with ODBC support
# SQL Server (MSSQL)
SQL Server datasources use the ODBC driver and Microsoft ODBC Driver 18 for SQL Server.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on SQL Server connections.
## Connection configuration
### URI format
```
mssql://user:password@host:1433/database
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | ------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"mssql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | SQL Server connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (SQL Server default `1433`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Optional fields
| Field | Type | Default | Description |
| ------------ | ------ | ------- | ------------------------------------------------------- |
| `encryption` | string | | TLS encryption mode (e.g., `"yes"`, `"no"`, `"strict"`) |
### Environment variables
| Variable | Description |
| ------------------- | ----------------------------------------------------------------------------------------------- |
| `ODBC_MSSQL_DRIVER` | Override the ODBC driver name (default: `"ODBC Driver 18 for SQL Server"`) |
| `GODEBUG` | Set to `x509negativeserial=1` for servers using older certificates with negative serial numbers |
The `ODBC_MSSQL_DRIVER` environment variable allows you to use a different version of the Microsoft ODBC driver (e.g., Driver 17) without modifying the connection configuration.
### Example connection
```json theme={null}
{
"name": "mssql-market-data",
"connection_type": "mssql",
"connection_sub_type": "database",
"uri": "mssql://libuser:s3cureP@ss@db.example.com:1433/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss",
"encryption": "yes"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "mssql-trades",
"connection_type": "mssql",
"connection_sub_type": "database",
"uri": "mssql://libuser:s3cureP@ss@sqlserver.example.com:1433/trades_db",
"server": "sqlserver.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss",
"encryption": "yes"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "mssql-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "mssql-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native SQL Server `datetime2` or `datetimeoffset` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## Encryption configuration
SQL Server connections support several TLS encryption modes via the `encryption` field:
| Value | Description |
| ---------- | --------------------------------------------- |
| `"yes"` | Require encrypted connection (recommended) |
| `"no"` | Do not encrypt the connection |
| `"strict"` | Strict TLS 1.2+ encryption (SQL Server 2022+) |
When using `encryption: "yes"` with self-signed certificates, you may need to add `TrustServerCertificate=yes` to the connection. For production, always use properly signed certificates.
## Legacy certificate support
For SQL Server instances using older certificates with negative serial numbers:
```bash theme={null}
export GODEBUG=x509negativeserial=1
```
This is common with older SQL Server installations that have not rotated their TLS certificates.
## Driver and dependencies
| Component | Details |
| ---------------- | ------------------------------------------------ |
| Driver | `ODBC driver` with ODBC Driver 18 for SQL Server |
| Protocol | ODBC / TDS |
| Arrow conversion | High-performance ODBC driver |
| ODBC required | Yes (Microsoft ODBC Driver 18) |
| Default port | 1433 |
The Microsoft ODBC Driver 18 must be installed on the CloudQuant Data Liberator host. To use a different driver version, set the `ODBC_MSSQL_DRIVER` environment variable.
## Troubleshooting
### ODBC driver not found
Verify the Microsoft ODBC Driver 18 is installed and registered:
```bash theme={null}
# List installed ODBC drivers
odbcinst -q -d
# Expected output should include:
# [ODBC Driver 18 for SQL Server]
```
Install the driver if missing:
```bash Debian/Ubuntu theme={null}
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/22.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update && ACCEPT_EULA=Y apt-get install -y msodbcsql18
```
```bash RHEL/CentOS theme={null}
curl https://packages.microsoft.com/config/rhel/8/prod.repo > /etc/yum.repos.d/mssql-release.repo
ACCEPT_EULA=Y yum install -y msodbcsql18
```
### Using a different driver version
Override the default driver with an environment variable:
```bash theme={null}
export ODBC_MSSQL_DRIVER="ODBC Driver 17 for SQL Server"
```
### Certificate errors (x509)
If you see `x509: certificate signed by unknown authority` or negative serial number errors:
```bash theme={null}
export GODEBUG=x509negativeserial=1
```
### Connection timeout
SQL Server may be configured to listen on a non-default port or require a named instance. Verify connectivity:
```bash theme={null}
# Test port connectivity
nc -zv sqlserver.example.com 1433
# For named instances, use the SQL Browser port (1434/UDP) or specify the port directly
```
### Windows authentication
CloudQuant Data Liberator uses SQL Server authentication (username/password). Windows/Kerberos authentication is not supported through this connection type.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed.
* For partitioned tables, use `dt_regex` to scope queries to relevant partitions.
* Consider using `datetime2` over `datetime` for better precision and performance.
# MySQL
Source: https://knowledge.cloudquant.com/datasource-config/mysql
Configure MySQL datasources with ODBC support
# MySQL
MySQL datasources use the ODBC driver and the MariaDB ODBC connector. The connection string is built internally from the provided fields.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on MySQL connections.
## Connection configuration
### URI format
```
mysql://user:password@host:3306/database
```
The URI is used as a reference identifier. CloudQuant Data Liberator builds the actual ODBC connection string internally from the individual connection fields (`server`, `database`, `username`, `password`).
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | -------------------------------------------------------------------- |
| `connection_type` | string | Must be `"mysql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | MySQL connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (MySQL default `3306`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Optional fields
| Field | Type | Default | Description |
| -------------- | ------ | --------- | ------------------------------------------------------------ |
| `ssl_disabled` | string | `"False"` | Set to `"True"` to disable SSL. Values: `"True"` / `"False"` |
### Example connection
```json theme={null}
{
"name": "mysql-market-data",
"connection_type": "mysql",
"connection_sub_type": "database",
"uri": "mysql://libuser:s3cureP@ss@db.example.com:3306/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss",
"ssl_disabled": "False"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "mysql-trades",
"connection_type": "mysql",
"connection_sub_type": "database",
"uri": "mysql://libuser:s3cureP@ss@db.example.com:3306/trades_db",
"server": "db.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss",
"ssl_disabled": "False"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "mysql-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "mysql-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native MySQL `DATETIME` or `TIMESTAMP` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## SSL configuration
By default, SSL is enabled. To disable SSL (e.g., for local development):
```json theme={null}
{
"ssl_disabled": "True"
}
```
Disabling SSL in production is not recommended. Always use SSL when connecting over untrusted networks. The MariaDB ODBC driver supports TLS 1.2+ by default.
## Driver and dependencies
| Component | Details |
| ---------------- | ------------------------------------ |
| Driver | `ODBC driver` with MariaDB ODBC 3.2+ |
| Protocol | ODBC |
| Arrow conversion | High-performance ODBC driver |
| ODBC required | Yes (MariaDB ODBC connector) |
| Default port | 3306 |
The MariaDB ODBC 3.2+ driver must be installed on the CloudQuant Data Liberator host. This driver is compatible with both MySQL and MariaDB servers. Verify installation with `odbcinst -q -d`.
## Troubleshooting
### ODBC driver not found
If you see errors about missing ODBC drivers, verify the MariaDB ODBC driver is installed and registered:
```bash theme={null}
# List installed ODBC drivers
odbcinst -q -d
# Expected output should include:
# [MariaDB ODBC 3.2 Driver]
```
Install the driver if missing:
```bash Debian/Ubuntu theme={null}
apt-get install libmariadb3 odbc-mariadb
```
```bash RHEL/CentOS theme={null}
yum install MariaDB-connector-odbc
```
### Connection refused
Verify the MySQL server is accepting connections on the specified host and port. Check that the `bind-address` in the MySQL configuration allows remote connections.
```bash theme={null}
mysql -h db.example.com -P 3306 -u libuser -p marketdata
```
### Authentication failed
Ensure the user has been granted access from the CloudQuant Data Liberator host's IP address:
```sql theme={null}
GRANT SELECT ON trades_db.* TO 'libuser'@'liberator-host' IDENTIFIED BY 's3cureP@ss';
FLUSH PRIVILEGES;
```
### SSL handshake errors
If SSL connections fail, check that the MySQL server's SSL certificate is valid and trusted. Set `ssl_disabled` to `"True"` temporarily to confirm the issue is SSL-related.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in MySQL.
* For partitioned tables, ensure `dt_regex` is correctly scoping queries to relevant partitions.
# Oracle
Source: https://knowledge.cloudquant.com/datasource-config/oracle
Configure Oracle Database datasources with Oracle database driver thin mode
# Oracle
Oracle datasources use the `Oracle database driver` in thin mode. No Oracle Instant Client installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on Oracle connections.
## Connection configuration
### URI format
```
oracle://user:password@host:1521/service_name
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | --------------------------------------------------------------------- |
| `connection_type` | string | Must be `"oracle"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | Oracle connection string |
| `server` | string | Database host address |
| `database` | string | Oracle service name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (Oracle default `1521`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`). Missing tables return a clear error instead of an empty sample.
### Optional fields
| Field | Type | Default | Description |
| ----------- | ------ | ------- | --------------------------------------------- |
| `auth_mode` | string | | Oracle authentication mode (e.g., `"SYSDBA"`) |
| `schema` | string | | Oracle schema name to query against |
### Environment variables
| Variable | Description |
| ----------------------- | ----------------------------------------------- |
| `ORACLE_DEBUG_BOUNDARY` | Enable debug timing for Oracle query boundaries |
### Example connection
```json theme={null}
{
"name": "oracle-market-data",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://libuser:s3cureP@ss@db.example.com:1521/ORCL",
"server": "db.example.com",
"database": "ORCL",
"username": "libuser",
"password": "s3cureP@ss"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database/service name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "oracle-trades",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://libuser:s3cureP@ss@oradb.example.com:1521/TRADEDB",
"server": "oradb.example.com",
"database": "TRADEDB",
"username": "libuser",
"password": "s3cureP@ss",
"schema": "MARKET_DATA"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "oracle-trades",
"data_args": {
"table_options": [
{
"db": "TRADEDB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "oracle-trades",
"batch_size": 10000,
"data_dt_column": "TRADE_TIME",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "SYMBOL"
},
"schema": [
{ "name": "SYMBOL", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "TRADE_TIME", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "PRICE", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "VOLUME", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "EXCHANGE", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Oracle column names are uppercase by default. Use uppercase names in your schema definition and `data_args` fields to match Oracle's metadata.
## Authentication modes
The `auth_mode` field supports Oracle authentication modes for privileged connections:
```json theme={null}
{
"name": "oracle-admin",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://sys:adminP@ss@db.example.com:1521/ORCL",
"server": "db.example.com",
"database": "ORCL",
"username": "sys",
"password": "adminP@ss",
"auth_mode": "SYSDBA"
}
```
Using `SYSDBA` authentication in production datasource connections is not recommended. Create a dedicated read-only user with `SELECT` privileges on the required tables.
## Schema-qualified tables
When the `schema` field is set on the connection, CloudQuant Data Liberator queries tables within that schema. This is useful when the database user's default schema differs from where the data resides:
```json theme={null}
{
"schema": "MARKET_DATA",
"table_options": [
{
"db": "TRADEDB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
]
}
```
This queries `MARKET_DATA.DAILY_TRADES` rather than relying on the user's default schema.
## Microsecond timestamp precision
Oracle `TIMESTAMP(6)` columns support microsecond precision. A custom-patched `oracledb 3.4.0` is available for CloudQuant Data Liberator that preserves full microsecond precision during data extraction.
The standard `Oracle database driver 3.3+` driver supports microsecond timestamps. The custom-patched 3.4.0 build provides additional optimizations for high-frequency timestamp extraction. Contact your CloudQuant Data Liberator administrator for the patched driver.
## Driver and dependencies
| Component | Details |
| ---------------------- | ----------------------------------------- |
| Driver | `Oracle database driver` 3.3+ (thin mode) |
| Protocol | Oracle Net (TNS) |
| Arrow conversion | Via Oracle driver native conversion |
| Oracle Client required | No (thin mode) |
| Default port | 1521 |
Thin mode connects directly to Oracle without requiring Oracle Instant Client. This simplifies deployment and eliminates native library dependencies.
## Troubleshooting
### Connection refused
Verify the Oracle listener is running and accepting connections on the specified host and port:
```bash theme={null}
# Test port connectivity
nc -zv oradb.example.com 1521
# Test with sqlplus (if available)
sqlplus libuser/s3cureP@ss@oradb.example.com:1521/TRADEDB
```
### ORA-12514: TNS listener does not currently know of service
The service name in the URI does not match any registered service on the Oracle listener. Verify the service name:
```sql theme={null}
-- On the Oracle server
SELECT name FROM v$services;
```
### ORA-01017: invalid username/password
Ensure the username and password are correct. Oracle passwords are case-sensitive (depending on the `SEC_CASE_SENSITIVE_LOGON` parameter).
### Table not found (ORA-00942)
Check that the user has `SELECT` privileges on the target table and that the schema is correctly specified:
```sql theme={null}
GRANT SELECT ON MARKET_DATA.DAILY_TRADES TO libuser;
```
### Debug timing
To enable debug timing for Oracle query boundaries, set the environment variable:
```bash theme={null}
export ORACLE_DEBUG_BOUNDARY=1
```
This logs timing information for each query phase, useful for diagnosing performance bottlenecks.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in Oracle.
* For partitioned tables, verify that Oracle partition pruning is being used by checking the execution plan.
* Use `TIMESTAMP(6)` columns for microsecond precision rather than `DATE` (which only supports second precision).
# Datasource Configuration Overview
Source: https://knowledge.cloudquant.com/datasource-config/overview
Guide to configuring datasource connections in CloudQuant Data Liberator
# Datasource configuration
CloudQuant Data Liberator supports a wide range of datasource types for ingesting time series data. Each datasource requires a **connection** (how to reach the data) and a **dataset** (what data to extract and how to interpret it).
See [Supported Data Formats](/datasource-config/supported-formats) for the canonical list of file extensions and data source categories Liberator can ingest, including formats added after 2.0.
## Supported datasource types
### File-based sources
| Type | Description |
| --------------------------------------------------------------- | ------------------------------------------------------------------- |
| [Local File (CSV/TSV)](/datasource-config/local-file) | Flat files on local/mounted storage |
| [S3](/datasource-config/s3) | Amazon S3 or S3-compatible object storage |
| [Azure Blob Storage](/datasource-config/azure-blob) | Microsoft Azure Blob containers |
| [SharePoint / OneDrive](/datasource-config/sharepoint-onedrive) | Microsoft 365 file storage via share link (2.2+); wildcards in 2.3+ |
| [SFTP](/datasource-config/sftp) | SSH File Transfer Protocol servers |
| [FTPS](/datasource-config/ftps) | FTP over TLS/SSL |
| [CIFS/SMB](/datasource-config/cifs) | Windows/Samba network file shares |
### Database sources
| Type | Description |
| ------------------------------------------- | -------------------------------------- |
| [PostgreSQL](/datasource-config/postgresql) | High-performance native driver |
| [MySQL](/datasource-config/mysql) | Via ODBC driver (MySQL-compatible) |
| [SQL Server](/datasource-config/mssql) | Via ODBC driver (ODBC Driver 18) |
| [Oracle](/datasource-config/oracle) | Via Oracle database driver (thin mode) |
| [Snowflake](/datasource-config/snowflake) | High-performance native driver |
### API sources
| Type | Description |
| ------------------------------------- | ---------------------------------- |
| [HubSpot](/datasource-config/hubspot) | HubSpot CRM objects via API (2.3+) |
### File and data formats
Beyond the delimited text examples in each connection guide, Liberator also ingests Parquet, Arrow IPC, Excel, XML, HDF5, ZIP archives, database tables, and API payloads. PSV (2.1+) and PCAP/PCAPng FIX capture (2.2+) are documented in [Supported Data Formats](/datasource-config/supported-formats).
## Architecture: connection + dataset
Every datasource in CloudQuant Data Liberator is composed of two parts:
### Connection
Defines **how to reach** the data — credentials, endpoints, paths, and transport protocol.
```
Connection → "Where is the data and how do I authenticate?"
```
### Dataset
Defines **what to extract** — which table/files, timestamp columns, key columns, schema, and data frequency.
```
Dataset → "What data do I want and how do I interpret it?"
```
In Liberator **2.3** and later, you can create both through the **AI Assisted** Create Dataset flow. Describe the source in plain language (or paste connection details), then review the auto-filled wizard steps before creating. See [AI-driven dataset onboarding](/administration/ai-dataset-onboarding).
## Common configuration concepts
### Timestamp configuration
All datasources require timestamp configuration to map source data into CloudQuant Data Liberator's microsecond timestamp (`muts`) format:
| Field | Description |
| ------------------ | ----------------------------------------------------------------- |
| `data_dt_column` | Column(s) containing the datetime |
| `data_dt_format` | Format string or parsing specification |
| `data_dt_timezone` | Timezone of the source data (e.g., `"UTC"`, `"America/New_York"`) |
| `data_dt_nudge` | Microsecond offset applied to timestamps |
#### Supported datetime formats
| Format | Description |
| --------------------- | ---------------------------------------------- |
| `"%Y-%m-%d %H:%M:%S"` | Standard strptime format |
| `"datetime"` | Native database datetime column |
| `"date"` | Native date column (date32/date64) |
| `"muts"` | Unix epoch microseconds |
| `"uts"` | Unix epoch seconds |
| `"nuts"` | Unix epoch nanoseconds |
| `true` | Auto-detect native datetime (database sources) |
### Key column configuration
The `data_key_column` field defines the symbol/key used for filtering queries:
```python theme={null}
# Simple string
"data_key_column": "symbol"
# Composite key with literals and columns
"data_key_column": [
{"type": "column", "value": "exchange"},
{"type": "literal", "value": "_"},
{"type": "column", "value": "ticker"}
]
```
### Schema definition
Each column in a dataset schema requires:
```json theme={null}
{
"name": "column_name",
"type": "int64",
"group": "value",
"description": "Human-readable description",
"display_name": "Display Name"
}
```
**Column types:** `string`, `int64`, `uint64`, `double`, `float`, `bool`, `date32`, `date64`, `time64`
**Column groups:**
* `key` — Symbol/key columns
* `time` — Timestamp columns
* `value` — Data columns
* `meta` — System columns (`_seq`, `muts`, etc.)
### Auto-generated columns
CloudQuant Data Liberator automatically generates these columns if not present in source data:
| Column | Type | Description |
| ----------- | ------ | -------------------------------------------- |
| `_seq` | uint64 | Sequential row number within partition |
| `muts` | int64 | Microseconds since Unix epoch |
| `timestamp` | string | Human-readable timestamp (America/New\_York) |
| `symbol` | string | Key column (copied from `data_key_column`) |
### File name date extraction
For file-based sources, dates can be extracted from filenames:
| Field | Description | Example |
| ------------------------- | --------------------------------------- | ------------------------------- |
| `fname_dt_regex` | Regex to match date portion of filename | `data_(\d{4}-\d{2}-\d{2})\.csv` |
| `fname_dt_format` | strptime format for the matched portion | `%Y-%m-%d` |
| `fname_dt_timezone` | Timezone of the filename date | `UTC` |
| `fname_dt_nudge` | Microsecond offset | `0` |
| `fname_dt_approx_seconds` | Approximate seconds per file | `86400` |
# PostgreSQL
Source: https://knowledge.cloudquant.com/datasource-config/postgresql
Configure PostgreSQL datasources with high-performance native driver support
# PostgreSQL
PostgreSQL datasources use a high-performance native driver for optimized data transfer. No ODBC installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on PostgreSQL connections.
## Connection configuration
### URI format
```
postgresql://user:password@host:5432/database
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | ------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"postgresql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | PostgreSQL connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (PostgreSQL default `5432`) |
The native driver communicates with the database using the PostgreSQL binary protocol and produces data batches directly, avoiding row-by-row conversion overhead. In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Example connection
```json theme={null}
{
"name": "postgres-market-data",
"connection_type": "postgresql",
"connection_sub_type": "database",
"uri": "postgresql://libuser:s3cureP@ss@db.example.com:5432/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "postgres-trades",
"connection_type": "postgresql",
"connection_sub_type": "database",
"uri": "postgresql://libuser:s3cureP@ss@db.example.com:5432/trades_db",
"server": "db.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "postgres-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "postgres-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native PostgreSQL `timestamp` or `timestamptz` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## Partitioned table example
For tables partitioned by date (e.g., `trades_2024_01`, `trades_2024_02`):
```json theme={null}
{
"table_options": [
{
"db": "trades_db",
"table_like": "trades_%",
"token": "trades",
"zone": "America/New_York",
"dt_regex": "trades_(\\d{4})_(\\d{2})",
"dt_format": "%Y_%m"
}
]
}
```
## Driver and dependencies
| Component | Details |
| --------------- | -------------------------- |
| Driver | PostgreSQL native driver |
| Protocol | PostgreSQL binary protocol |
| Data conversion | Native (high-performance) |
| ODBC required | No |
| Default port | 5432 |
The native driver is bundled with CloudQuant Data Liberator. No additional driver installation is needed on the host system.
## Troubleshooting
### Connection refused
Verify the PostgreSQL server is accepting connections on the specified host and port. Check `pg_hba.conf` for client authentication rules.
```bash theme={null}
psql -h db.example.com -p 5432 -U libuser -d marketdata
```
### Authentication failed
Ensure the `username` and `password` in the connection match a valid PostgreSQL role. Check that the role has `CONNECT` privilege on the database and `SELECT` privilege on the target tables.
### SSL/TLS errors
If the server requires SSL, append `?sslmode=require` to the URI:
```
postgresql://libuser:s3cureP@ss@db.example.com:5432/marketdata?sslmode=require
```
Using `sslmode=disable` in production is not recommended. Always use `sslmode=require` or `sslmode=verify-full` when connecting over untrusted networks.
### Empty results from partitioned tables
Verify that `dt_regex` correctly matches the table name suffix and that `dt_format` corresponds to the captured groups. Test the regex against your actual table names.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in PostgreSQL.
* Use `table_like` to target specific tables rather than scanning all partitions.
# Amazon S3
Source: https://knowledge.cloudquant.com/datasource-config/s3
Configure S3 or S3-compatible object storage datasources
# Amazon S3
S3 datasources allow CloudQuant Data Liberator to read CSV, TSV, and Parquet files directly from Amazon S3 buckets or S3-compatible object storage services (MinIO, Wasabi, Backblaze B2, etc.).
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest on S3, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------------- | ------ | --------------------------------------------------- |
| `connection_type` | string | Must be `"s3"` |
| `aws_access_key_id` | string | AWS access key ID |
| `aws_secret_access_key` | string | AWS secret access key |
| `bucket` | string | S3 bucket name |
| `endpoint` | string | S3 endpoint URL, e.g., `"https://s3.amazonaws.com"` |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ------ | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `prefix` | string | `""` | Key prefix (virtual directory) within the bucket |
| `request_style` | string | auto (2.4+) | S3 addressing: `"path"` or `"virtual"`. In Liberator **2.4** and later, Test Connection probes the endpoint and persists the correct style. Override only if the probe is wrong. |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
| `object_key` | string | | Object key pattern for file selection |
In Liberator **2.4** and later, Test Connection auto-detects path-style vs virtual-hosted addressing for AWS and S3-compatible endpoints (MinIO, Ceph, Wasabi, Oracle Cloud, and custom hosts). You can still set `request_style` explicitly. For older releases, set `request_style` to `"path"` for most S3-compatible services and keep `"virtual"` for AWS S3.
### Example connection
```json theme={null}
{
"name": "s3-market-data",
"connection_type": "s3",
"aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
"aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"bucket": "my-market-data",
"endpoint": "https://s3.amazonaws.com",
"prefix": "daily/equities/",
"request_style": "virtual"
}
```
Never commit AWS credentials to version control. Use environment variables or a secrets manager to inject credentials at deployment time.
## CSV/TSV dataset
The `data_args` are identical to [Local File](/datasource-config/local-file) sources. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
### Complete CSV example
```json theme={null}
{
"name": "s3-equity-trades",
"connection": "s3-market-data",
"data_args": {
"file_pattern": "trades_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "timestamp",
"data_dt_format": "%Y-%m-%d %H:%M:%S.%f",
"data_dt_timezone": "UTC",
"data_key_column": "symbol",
"fname_dt_regex": "trades_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "UTC",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "timestamp", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "size", "type": "int64", "group": "value", "description": "Trade size" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
## Parquet dataset
This provides passthrough access to Parquet data without intermediate caching, leveraging Arrow's native Parquet reader.
Parquet files offer zero-copy reads and columnar pushdown. This is significantly more efficient than converting Parquet to CSV.
### Complete Parquet example
```json theme={null}
{
"name": "s3-equity-bars-parquet",
"connection": "s3-market-data",
"data_args": {
"file_pattern": "bars_*.parquet",
"data_dt_column": "bar_time",
"data_dt_format": "datetime",
"data_dt_timezone": "America/New_York",
"data_key_column": "symbol",
"fname_dt_regex": "bars_(\\d{4}-\\d{2}-\\d{2})\\.parquet",
"fname_dt_format": "%Y-%m-%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "bar_time", "type": "string", "group": "time", "description": "Bar timestamp" },
{ "name": "open", "type": "double", "group": "value", "description": "Open price" },
{ "name": "high", "type": "double", "group": "value", "description": "High price" },
{ "name": "low", "type": "double", "group": "value", "description": "Low price" },
{ "name": "close", "type": "double", "group": "value", "description": "Close price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Bar volume" }
]
}
```
## S3-compatible storage
### MinIO example
```json theme={null}
{
"name": "minio-connection",
"connection_type": "s3",
"aws_access_key_id": "minio-access-key",
"aws_secret_access_key": "minio-secret-key",
"bucket": "market-data",
"endpoint": "https://minio.internal.example.com:9000",
"request_style": "path"
}
```
Most S3-compatible services require `request_style` set to `"path"`. Only AWS S3 defaults to virtual-hosted style.
## IAM permissions
The IAM user or role associated with the access key needs at minimum:
* `s3:GetObject` on the bucket objects
* `s3:ListBucket` on the bucket
See the [S3 Bucket Setup](/integrations/s3-bucket-setup) guide for detailed IAM policy configuration.
# SFTP
Source: https://knowledge.cloudquant.com/datasource-config/sftp
Configure SFTP datasources
# SFTP
SFTP (SSH File Transfer Protocol) datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from remote servers over an encrypted SSH connection. CloudQuant Data Liberator mounts the remote directory via SSHFS/FUSE.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest over SFTP, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | ---------------------------------- |
| `connection_type` | string | Must be `"sftp"` |
| `host` | string | SFTP server hostname or IP address |
| `user` | string | Username for authentication |
You must provide either `password` or `key` for authentication. If both are specified, key-based authentication takes precedence.
### Optional fields
| Field | Type | Default | Description |
| ------------- | ------ | ------- | ----------------------------------------------------------------- |
| `port` | int | `22` | SSH port number |
| `password` | string | | Password for password-based authentication |
| `key` | string | | SSH private key content (PEM format) for key-based authentication |
| `prefix` | string | `""` | Remote directory path to use as root |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
### Example connection (password authentication)
```json theme={null}
{
"name": "sftp-vendor-data",
"connection_type": "sftp",
"host": "sftp.vendor.example.com",
"port": 22,
"user": "datauser",
"password": "s3cur3P@ssw0rd",
"prefix": "/data/daily-feeds/"
}
```
### Example connection (key authentication)
```json theme={null}
{
"name": "sftp-internal-data",
"connection_type": "sftp",
"host": "data-server.internal.net",
"port": 2222,
"user": "liberator-svc",
"key": "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNza...\n-----END OPENSSH PRIVATE KEY-----",
"prefix": "/exports/market-data/"
}
```
Avoid embedding private keys or passwords directly in configuration files. Use environment variables or a secrets manager to inject credentials at deployment time.
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "sftp-trades-feed",
"connection_type": "sftp",
"host": "sftp.dataprovider.com",
"port": 22,
"user": "cq-ingest",
"password": "vendorPassword123",
"prefix": "/feeds/trades/"
}
```
### Dataset
```json theme={null}
{
"name": "vendor-trades",
"connection": "sftp-trades-feed",
"data_args": {
"file_pattern": "trades_*.csv.gz",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": ["date", "time"],
"data_dt_format": ["%Y%m%d", "%H:%M:%S.%f"],
"data_dt_timezone": "America/New_York",
"data_key_column": "symbol",
"fname_dt_regex": "trades_(\\d{8})\\.csv\\.gz",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "date", "type": "string", "group": "time", "description": "Trade date" },
{ "name": "time", "type": "string", "group": "time", "description": "Trade time" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "size", "type": "int64", "group": "value", "description": "Trade size" },
{ "name": "condition", "type": "string", "group": "value", "description": "Sale condition code" }
]
}
```
CloudQuant Data Liberator supports reading gzip-compressed files (`.csv.gz`) transparently. Use compressed files on SFTP connections to reduce transfer time over slow or high-latency links.
## Network requirements
Ensure the following network connectivity from the CloudQuant Data Liberator host:
| Requirement | Detail |
| ------------------ | ------------------------------------------------------------------ |
| **Outbound port** | TCP port 22 (or custom port) to the SFTP server |
| **DNS resolution** | The hostname must resolve from the CloudQuant Data Liberator host |
| **Firewall rules** | Whitelist the CloudQuant Data Liberator host IP on the SFTP server |
| **SSH host key** | The server's host key must be trusted (added to known\_hosts) |
# SharePoint and OneDrive
Source: https://knowledge.cloudquant.com/datasource-config/sharepoint-onedrive
Connect SharePoint or OneDrive data sources using a Microsoft share link
# SharePoint and OneDrive
SharePoint and OneDrive connections let Liberator read delimited files, Parquet, Excel, and other [supported formats](/datasource-config/supported-formats) from Microsoft 365 storage.
In Liberator 2.2 and later, paste a standard **share link** copied from the browser or desktop to connect. In Liberator **2.3** and later, wildcard folder and file patterns (for example, `reports/2026-*/trades_*.csv`) are supported for both SharePoint and OneDrive sources.
## Connect with a share link
In the browser or desktop app, open the file or folder, then **Share → Copy link**. The link looks like `https://your-org.sharepoint.com/:f:/s/...`.
Go to **Connections → Add Connection** and select **SharePoint** or **OneDrive**.
Paste the link into the **Share link** field. Liberator clears any previously entered folder path when a share link is detected.
Complete the **Sign in with Microsoft** dialog. If it does not appear automatically, click **Sign in with Microsoft**.
Click **Test Connection**. On success, the resolved internal path replaces the share link in the field. Confirm the account has at least read access to the linked location.
Continue through dataset configuration (file pattern, columns, date/time) as for other file-based sources.
## Fixed date and time (non-time-series files)
For static reference files without a date column:
1. On the **Date / Time** step, select **Fixed date and time**.
2. Enter the timestamp Liberator should assign to records.
If the source has a date column, select it from the **Date column** dropdown instead.
## Date format picker (2.2+)
When choosing a date format during onboarding, Liberator:
* Shows a **live example from your sample data** (for example, `25 Jun 2026` instead of generic `DD MMM YYYY`).
* **Ranks formats by likelihood**, with the best match at the top.
Select the example that matches your source dates.
## Related
File extensions available on SharePoint and OneDrive connections
Shared `data_args` fields for file-based datasets
# Snowflake
Source: https://knowledge.cloudquant.com/datasource-config/snowflake
Configure Snowflake datasources with native high-performance driver
# Snowflake
Snowflake datasources use the Snowflake native driver for high-performance native data transfer. No ODBC installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on Snowflake connections.
## Connection configuration
### URI format
```
snowflake://user:password@account/database/schema?warehouse=compute_wh&role=accountadmin
```
### Required fields
| Field | Type | Description |
| --------------------- | ------ | --------------------------------------------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"snowflake"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | Snowflake connection string |
| `server` | string | Snowflake account URL (e.g., `"xy12345.us-east-1.snowflakecomputing.com"`) |
| `database` | string | Snowflake database name |
| `username` | string | Snowflake user |
| `password` | string | Snowflake password. In Liberator **2.4+**, special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`) are supported |
| `account` | string | Snowflake account identifier (e.g., `"xy12345.us-east-1"`) |
| `warehouse` | string | Compute warehouse name |
| `role` | string | Snowflake role |
### Optional fields
| Field | Type | Default | Description |
| -------- | ------ | ------- | --------------------- |
| `schema` | string | | Snowflake schema name |
### Environment variables
| Variable | Description |
| --------------------- | ------------------------------------------------------ |
| `SNOWFLAKE_DEBUGGING` | Enable verbose debug logging for Snowflake connections |
### Example connection
```json theme={null}
{
"name": "snowflake-market-data",
"connection_type": "snowflake",
"connection_sub_type": "database",
"uri": "snowflake://libuser:s3cureP@ss@xy12345.us-east-1/MARKETDATA/PUBLIC?warehouse=COMPUTE_WH&role=DATA_READER",
"server": "xy12345.us-east-1.snowflakecomputing.com",
"database": "MARKETDATA",
"username": "libuser",
"password": "s3cureP@ss",
"account": "xy12345.us-east-1",
"warehouse": "COMPUTE_WH",
"role": "DATA_READER",
"schema": "PUBLIC"
}
```
The Snowflake native driver uses Snowflake's native result set format. Data is returned as batches directly from the Snowflake service with no row-by-row conversion.
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "snowflake-trades",
"connection_type": "snowflake",
"connection_sub_type": "database",
"uri": "snowflake://libuser:s3cureP@ss@xy12345.us-east-1/TRADES_DB/MARKET?warehouse=COMPUTE_WH&role=accountadmin",
"server": "xy12345.us-east-1.snowflakecomputing.com",
"database": "TRADES_DB",
"username": "libuser",
"password": "s3cureP@ss",
"account": "xy12345.us-east-1",
"warehouse": "COMPUTE_WH",
"role": "accountadmin",
"schema": "MARKET"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "snowflake-trades",
"data_args": {
"table_options": [
{
"db": "TRADES_DB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "snowflake-trades",
"batch_size": 10000,
"data_dt_column": "TRADE_TIME",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "SYMBOL"
},
"schema": [
{ "name": "SYMBOL", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "TRADE_TIME", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "PRICE", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "VOLUME", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "EXCHANGE", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Snowflake identifiers are uppercase by default. Use uppercase names in your schema definition and `data_args` fields to match Snowflake's metadata, unless the objects were created with double-quoted lowercase names.
## Warehouse and role configuration
The `warehouse` and `role` fields control compute resources and access permissions:
### Warehouse
The warehouse determines the compute cluster used to execute queries. Choose a warehouse sized appropriately for your query workload:
```json theme={null}
{
"warehouse": "COMPUTE_WH"
}
```
Ensure the specified warehouse is set to auto-resume, or queries will fail when the warehouse is suspended. CloudQuant Data Liberator does not issue `ALTER WAREHOUSE ... RESUME` commands.
### Role
The role determines which database objects are accessible:
```json theme={null}
{
"role": "DATA_READER"
}
```
Create a dedicated read-only role for CloudQuant Data Liberator connections rather than using `accountadmin`. Grant `SELECT` on the required databases and schemas to this role.
## Debug logging
Enable verbose debug logging for Snowflake connections to diagnose connectivity or query issues:
```bash theme={null}
export SNOWFLAKE_DEBUGGING=1
```
This produces detailed logs including connection negotiation, query execution timing, and data batch transfer metrics.
## Driver and dependencies
| Component | Details |
| --------------- | --------------------------------------- |
| Driver | `Snowflake native driver` |
| Protocol | Snowflake REST API / streaming protocol |
| Data conversion | Native (high-performance) |
| ODBC required | No |
| Default port | 443 (HTTPS) |
The Snowflake native driver is bundled with CloudQuant Data Liberator. No additional driver installation is needed on the host system. All communication uses HTTPS on port 443.
## Troubleshooting
### Account identifier format
The `account` field must include the region if your account is not in the default AWS us-west-2 region:
```
# US East (Ohio)
xy12345.us-east-1
# EU (Frankfurt)
xy12345.eu-central-1
# Azure (East US 2)
xy12345.east-us-2.azure
```
### Authentication failed (390100)
Verify the username, password, and account identifier. Snowflake passwords are case-sensitive. If multi-factor authentication (MFA) is enabled for the user, you may need to use a service account without MFA.
### Warehouse is suspended
If queries fail with a warehouse suspension error, ensure the warehouse is configured with `AUTO_RESUME = TRUE`:
```sql theme={null}
ALTER WAREHOUSE COMPUTE_WH SET AUTO_RESUME = TRUE;
```
### Role does not have access
Verify the role has the required privileges:
```sql theme={null}
GRANT USAGE ON DATABASE TRADES_DB TO ROLE DATA_READER;
GRANT USAGE ON SCHEMA TRADES_DB.MARKET TO ROLE DATA_READER;
GRANT SELECT ON ALL TABLES IN SCHEMA TRADES_DB.MARKET TO ROLE DATA_READER;
```
### Network connectivity
Snowflake connections require HTTPS (port 443) access to `*.snowflakecomputing.com`. Ensure firewalls and proxy servers allow this traffic.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure Snowflake clustering keys align with your datetime and key columns.
* Use an appropriately sized warehouse (e.g., `MEDIUM` or `LARGE`) for large datasets.
* Check that the warehouse is not queued behind other workloads using Snowflake's query history.
# Supported Data Formats
Source: https://knowledge.cloudquant.com/datasource-config/supported-formats
File and data formats CloudQuant Data Liberator can ingest, by platform version
# Supported data formats
CloudQuant Data Liberator ingests time-series and tabular data through **connections** (where data lives) and **datasets** (how files or tables are interpreted). The tables below list every **file extension** and **data source category** the platform understands.
Formats are selected automatically from your `file_pattern`, file extension, or connection type when you onboard through the Liberator UI. You rarely need to set anything manually.
This page is the canonical format reference. Each [connection guide](/datasource-config/overview) links here for the formats available on that storage type.
## Supported in version 2.0
These formats were available in the Liberator **2.0** release and remain supported on current versions.
### Delimited text files
| Extension | Delimiter | Configuration |
| --------- | --------------- | ----------------------------------------------------- |
| `.csv` | Comma (default) | `file_pattern` such as `*.csv` |
| `.tsv` | Tab | `file_pattern` such as `*.tsv` |
| `.txt` | Auto-detected | Same pipeline as CSV; delimiter inferred when omitted |
Set `sep_override` in `data_args` to force comma, tab, pipe (`\|`), or semicolon when auto-detection is not sufficient.
### Columnar and binary files
| Extension | Description |
| -------------------- | ------------------------------------------------- |
| `.parquet` | Apache Parquet; columnar reads with type pushdown |
| `.arrow`, `.feather` | Apache Arrow IPC (Feather v2) |
On S3, Parquet datasets can use **passthrough** mode (direct read without intermediate caching). See the [S3 Parquet example](/datasource-config/s3#parquet-dataset).
### Structured office and scientific files
| Extension | Description | Extra configuration |
| --------------- | ------------------------- | ---------------------------------------------------- |
| `.xlsx`, `.xls` | Microsoft Excel workbooks | First sheet sampled at onboarding |
| `.xml` | XML documents | `xml_args` for element paths (set during onboarding) |
| `.h5`, `.hdf5` | HDF5 scientific arrays | `h5py_groups` for dataset path inside the file |
### Archives
| Extension | Description |
| --------- | --------------------------------------------------------------------------------------- |
| `.zip` | ZIP archive; Liberator inspects the inner file and applies the matching format delegate |
### Database sources
| Connection type | Engine | Access pattern |
| --------------- | ------------------------------ | ------------------------- |
| PostgreSQL | Native high-performance driver | Table or view per dataset |
| MySQL | ODBC (MariaDB-compatible) | Table or view per dataset |
| SQL Server | ODBC Driver 18 | Table or view per dataset |
| Oracle | Thin driver | Table or view per dataset |
| Snowflake | Native driver | Table or view per dataset |
### API-backed datasets
REST endpoints that return JSON tabular payloads can be onboarded as **API** connections. The platform normalizes responses into the same query surface as file- and database-backed datasets.
### Schema column types
Regardless of source format, dataset schemas use these column types:
`string`, `int64`, `uint64`, `double`, `float`, `bool`, `date32`, `date64`, `time64`
See [Datasource configuration overview](/datasource-config/overview#schema-definition) for column groups (`key`, `time`, `value`, `meta`).
## Formats added after version 2.0
The following ingest formats were added in subsequent releases. They are available on current CloudQuant-managed environments at those versions and later.
### Version 2.1 — PSV (pipe-separated values)
| Extension | Delimiter | Notes |
| --------- | ----------- | ----------------------------------------------- |
| `.psv` | Pipe (`\|`) | First-class extension alongside CSV and Parquet |
PSV files use the same onboarding flow as CSV: header row, per-column type inference, and configurable null sentinel. You can also ingest pipe-delimited `.csv` or `.txt` files by setting `sep_override` to `"\|"` without renaming the file.
### Version 2.2 — PCAP / PCAPng (FIX tick capture)
| Extension | Description |
| --------- | ------------------------------ |
| `.pcap` | Classic packet capture |
| `.pcapng` | Next-generation packet capture |
Liberator extracts **FIX-protocol messages** from TCP payloads in packet captures and exposes them through the standard query API. Typical columns include FIX tags such as `35` (MsgType), `49` (SenderCompID), `52` (SendingTime), and `55` (Symbol), plus `_pcap_ts_ns` for the capture timestamp.
Use PCAP datasets when you capture exchange feeds at the wire level and want the same query model as historical bar or trade datasets.
## Added in version 2.3
No new file or packet-capture ingest formats ship in 2.3. Connectivity additions in this release (HubSpot CRM objects, OneDrive/SharePoint wildcards) are covered in the [HubSpot](/datasource-config/hubspot) and [SharePoint and OneDrive](/datasource-config/sharepoint-onedrive) guides and in [What's New in Liberator 2.3](/whats-new/liberator-2.3).
## Added in version 2.4
No new file or packet-capture ingest formats ship in 2.4. Query, catalog, and connection changes in this release (symbol pattern filters, dataset health cards, S3 request-style auto-detect, required database port) are covered in [What's New in Liberator 2.4](/whats-new/liberator-2.4).
## Where each format applies
All **file-based** formats in the tables above can be stored on any file-backed connection type:
| Connection | Guide |
| ----------------------------- | ------------------------------------------- |
| Local / mounted directory | [Local File](/datasource-config/local-file) |
| Amazon S3 (and S3-compatible) | [S3](/datasource-config/s3) |
| Azure Blob Storage | [Azure Blob](/datasource-config/azure-blob) |
| SFTP | [SFTP](/datasource-config/sftp) |
| FTPS | [FTPS](/datasource-config/ftps) |
| CIFS / SMB | [CIFS](/datasource-config/cifs) |
**Database** and **API** formats map to their respective connection guides:
| Connection | Guide |
| ---------- | ------------------------------------------- |
| PostgreSQL | [PostgreSQL](/datasource-config/postgresql) |
| MySQL | [MySQL](/datasource-config/mysql) |
| SQL Server | [SQL Server](/datasource-config/mssql) |
| Oracle | [Oracle](/datasource-config/oracle) |
| Snowflake | [Snowflake](/datasource-config/snowflake) |
## Choosing a format
| Use case | Recommended format |
| ---------------------------------------------------- | ------------------------------------- |
| Human-readable exports from spreadsheets or ETL jobs | CSV or TSV |
| Vendor pipe-delimited daily drops | PSV (2.1+) or CSV with `sep_override` |
| Large historical archives, column pruning | Parquet |
| Low-latency interchange between Arrow-native tools | Arrow IPC / Feather |
| Excel exports from business users | XLSX |
| Scientific simulation output | HDF5 |
| Hierarchical vendor XML feeds | XML |
| SQL warehouse tables already in your estate | Matching database connection |
| Wire-level FIX tick replay | PCAP / PCAPng (2.2+) |
## Related
Connection + dataset architecture and shared `data_args` fields
Full `data_args` reference for file-based sources
# Azure Blob Storage
Source: https://knowledge.cloudquant.com/integrations/azure-blob-storage
Retrieving access keys for Azure Blob Storage connections with CloudQuant Data Liberator
# Azure Blob Storage connections
This guide explains how to retrieve and configure Azure Blob Storage access keys for use with the CloudQuant Data Liberator platform.
## Prerequisites
* An Azure account with a Storage Account
* Access to the Azure Portal
* Your CloudQuant account credentials
## Retrieving access keys
### Step 1: Navigate to your storage account
1. Sign in to the [Azure Portal](https://portal.azure.com)
2. Navigate to **Storage accounts**
3. Select the storage account you want to connect
### Step 2: Find access keys
1. In the storage account menu, select **Security + networking** > **Access keys**
2. You will see two keys: **key1** and **key2**
3. Select **Show** to reveal the key values
### Step 3: Copy required information
You will need:
* **Storage account name** — The name of your Azure Storage Account
* **Access key** — Either key1 or key2
* **Container name** — The blob container holding your data
Keep your access keys secure. Never commit them to source control or share them in plain text.
## Configuring the connection
Provide the following to CloudQuant for CloudQuant Data Liberator connectivity:
```
Storage Account: your_storage_account_name
Access Key: your_access_key
Container: your_container_name
```
## Connection string format
If configuring programmatically, the connection string format is:
```
DefaultEndpointsProtocol=https;AccountName=your_account;AccountKey=your_key;EndpointSuffix=core.windows.net
```
## Key rotation
Azure provides two keys to enable rotation without downtime:
1. Update your applications to use **key2**
2. Regenerate **key1**
3. Update applications to use the new **key1**
4. Regenerate **key2**
Rotate your access keys regularly as a security best practice.
## Troubleshooting
| Issue | Solution |
| --------------------- | ----------------------------------------------------------------- |
| Authentication failed | Verify the access key hasn't been rotated |
| Container not found | Check the container name and ensure it exists |
| Network error | Verify firewall rules allow access from CloudQuant Data Liberator |
# ChatGPT
Source: https://knowledge.cloudquant.com/integrations/chatgpt
Connect ChatGPT to CloudQuant Data Liberator with the MCP connector and OAuth authentication
# Connecting Liberator to ChatGPT
Use ChatGPT's MCP connector with **OAuth** to query CloudQuant Data Liberator from a ChatGPT workspace. Each user authenticates with their own Liberator credentials, so data access follows individual entitlements rather than a shared API key.
## Prerequisites
* Access through [app.cloudquant.ai](https://app.cloudquant.ai) — ChatGPT MCP setup for Liberator is done from that CloudQuant environment.
* A **ChatGPT Pro or Plus** account — the MCP connector is only available on paid tiers.
* **Developer tool access** enabled in ChatGPT (required before you can create custom Apps). Enable it under **Settings**.
* A CloudQuant Data Liberator account with valid username and password.
* The **Liberator MCP Endpoint URL** from the Liberator UI (for example, `https://api.cloudquant.ai/mcp`).
OAuth is the recommended authentication method. Each user who connects through the ChatGPT app authenticates with their own Liberator credentials, so entitlements stay per-user.
## Setup steps
In ChatGPT, click your workspace or profile name in the lower-left corner, then select **Settings**.
In Settings, open the **Advanced** section, click **Apps**, then click **Create App**.
In the New App dialog, set:
| Field | Value |
| --------------- | ----------------------------------------------------------------------------------------------- |
| **Name** | `CloudQuant Liberator` |
| **Description** | `MCP Connector for CloudQuant Data Liberator` |
| **Connection** | Select **Server URL** (not Tunnel). Enter the Liberator MCP Endpoint URL from the Liberator UI. |
Example MCP Endpoint URL:
```text theme={null}
https://api.cloudquant.ai/mcp
```
Copy the exact URL from your Liberator UI if your environment differs.
Under **Authentication**, select **OAuth**. This is the recommended method for Liberator and keeps access tied to each user's credentials.
ChatGPT shows a warning that custom MCP servers introduce risk. Check **I understand and want to continue**, then click **Create**. The Create button stays inactive until you acknowledge the warning.
ChatGPT starts the OAuth flow and shows a **Grant Access to ChatGPT** screen. Approve access for:
* User profile
* `openid`
* User roles
* Email address
* `groups`
Click **Yes** to grant access so ChatGPT can authenticate to the Liberator MCP server on your behalf.
Once access is granted, the CloudQuant Liberator MCP connector is ready to use in your ChatGPT workspace.
## Using the connector
After the app is created, Liberator MCP tools are available in ChatGPT conversations:
| Tool | Purpose |
| -------------------- | ------------------------------------------------- |
| `list_datasets` | List available Liberator datasets |
| `get_dataset_config` | Fetch configuration and schema for a dataset |
| `query_dataset` | Query a Liberator dataset by name with parameters |
Because OAuth is used, each user who interacts with the connector is prompted to authenticate with their own CloudQuant credentials. Individual data access and entitlements apply — no shared API keys are required.
## Related
Add Liberator as an MCP tool in a Copilot Studio agent
Install the Liberator .mcpb extension in Claude Desktop
Liberator 2.3 release notes, including AI-driven reporting
Connect Liberator metrics to Grafana with a Bearer token
# Claude Desktop
Source: https://knowledge.cloudquant.com/integrations/claude
Connect Claude Desktop to CloudQuant Data Liberator with the MCP extension bundle
# Connecting Liberator to Claude Desktop
Install the Liberator extension bundle (`.mcpb`) in Claude Desktop to query Liberator datasets directly from chat through the Model Context Protocol (MCP).
## Prerequisites
* [Claude Desktop](https://claude.ai/download) installed on your machine
* A CloudQuant Data Liberator account
* The Liberator **MCP Endpoint URL** from the Liberator UI (for example, `https://api.cloudquant.ai/mcp`)
* The Liberator **`.mcpb` extension bundle**, downloadable from the Liberator UI **Profile → Documentation → A.I.** section (or from `https:///permissions/download-liberator-client/?lang=mcpb` while signed in)
## Setup steps
In the Liberator UI, open **Profile → Documentation → A.I. → Claude Desktop** and download the `.mcpb` file. Keep it somewhere easy to find.
In Claude Desktop, open **Settings → Extensions → Install Extension**.
Select the downloaded `.mcpb` file.
Set the extension **Server URL** to your Liberator MCP Endpoint URL from the Liberator UI, for example:
```text theme={null}
https://api.cloudquant.ai/mcp
```
Once installed, Claude can use Liberator MCP tools such as `list_datasets`, `get_dataset_config`, and `query_dataset` from chat.
## Related
Connect ChatGPT with the MCP connector and OAuth
Add Liberator as an MCP tool in a Copilot Studio agent
# Microsoft Copilot Studio
Source: https://knowledge.cloudquant.com/integrations/copilot
Connect Microsoft Copilot Studio to CloudQuant Data Liberator with MCP and OAuth 2.0
# Connecting Liberator to Microsoft Copilot Studio
Add Liberator as an MCP connector in a **Copilot Studio** agent using **OAuth 2.0**, so each user signs in with their own Liberator credentials. Configure this from within Copilot Studio (not Microsoft 365 or Teams). An existing agent is required.
## Prerequisites
* Access to [Microsoft Copilot Studio](https://copilotstudio.microsoft.com/) with permission to edit an agent
* An existing Copilot Studio agent
* A CloudQuant Data Liberator account with valid username and password
* The Liberator **MCP Endpoint URL** from the Liberator UI (for example, `https://api.cloudquant.ai/mcp`)
Requires **Streamable HTTP** transport and **OAuth 2.0 Dynamic Client Registration (DCR)**. The connector can be shared org-wide, but each user creates their own connection by signing in with their own Liberator credentials. No shared API keys are needed.
## Setup steps
In your agent, go to **Tools → New Tool** and select **Model Context Protocol (MCP)**.
Enter a server name and description, and set **Server URL** to your Liberator MCP Endpoint URL, for example:
```text theme={null}
https://api.cloudquant.ai/mcp
```
Under **Authentication**, select **OAuth 2.0** with **Type** set to **Dynamic discovery**, then click **Create**.
On the **Add tool** screen, open the **Connection** dropdown, select **Create new connection**, then **Create**.
Sign in with your Liberator username and password and grant access (profile, `openid`, email).
Click **Add and configure**. Copilot auto-registers the Liberator tools (`list_datasets`, `get_dataset_config`, `query_dataset`).
## Using the connector
After the tool is added, Liberator MCP tools are available to the agent:
| Tool | Purpose |
| -------------------- | ------------------------------------------------- |
| `list_datasets` | List available Liberator datasets |
| `get_dataset_config` | Fetch configuration and schema for a dataset |
| `query_dataset` | Query a Liberator dataset by name with parameters |
Each user who uses the connector signs in with their own Liberator credentials, so entitlements stay per-user.
## Related
Connect ChatGPT with the MCP connector and OAuth
Install the Liberator .mcpb extension in Claude Desktop
# Grafana
Source: https://knowledge.cloudquant.com/integrations/grafana
Connect your Grafana instance to CloudQuant Data Liberator's Prometheus stack using a rotatable Bearer token
# Grafana integration
CloudQuant Data Liberator exposes a curated, read-only slice of its internal Prometheus stack at a Bearer-authenticated endpoint. You can wire your own Grafana (or any Prometheus-API-compatible client) to the same metrics that power the in-product **System Monitoring** view.
This integration is intended for ops teams, on-call rotations, and infrastructure dashboards that need to live alongside metrics from systems outside CloudQuant.
## At a glance
| | |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Endpoint** | `https:///metrics-api-bearer/` |
| **Authentication** | `Authorization: Bearer cqm_<48-hex-chars>` |
| **Protocol** | Standard Prometheus HTTP API (`/api/v1/query`, `/api/v1/query_range`, `/api/v1/series`, `/api/v1/labels`, `/api/v1/label//values`, `/api/v1/status/buildinfo`, `/federate`) |
| **Surface** | Read-only. Admin endpoints (`/api/v1/admin/*`) and remote-write are gated off at the Prometheus server, not just at the gateway. |
| **Token management** | Liberator UI → **System Monitoring → Grafana Integration** tab |
## Prerequisites
* A user with **super-admin** privileges on your Liberator instance (token management is a super-admin operation).
* Network reachability from your Grafana host to your Liberator hostname over HTTPS.
* Grafana 9.x or newer (any version that supports custom HTTP headers on the Prometheus datasource).
## Step 1 — Issue a Bearer token
1. Sign in to the Liberator UI as a super-admin.
2. Open **System Monitoring** from the top navigation.
3. Click the **Grafana Integration** action in the upper-right of the page.
4. Click **Generate token**. The new token is shown **exactly once**, so copy it immediately.
Tokens follow the format `cqm_` + 48 hex characters. Every token issued through this dialog is logged with the issuing user and timestamp; tokens can be listed and revoked from the same dialog.
The full token value is only visible at issue time. Treat it like a password: store it in your secrets manager and never commit it to source control. If a token is lost, revoke it and issue a new one rather than trying to recover it.
Issue one token per logical consumer (e.g. `grafana-prod`, `federated-prometheus-eu`) so you can revoke a single consumer's access without disrupting others.
## Step 2 — Configure the Grafana datasource
### Through the Grafana UI
1. **Connections → Data sources → Add data source → Prometheus.**
2. Set the fields below:
| Field | Value |
| ------------------------- | ------------------------------------------------------- |
| **Name** | `cloudquant-liberator` (or any label you prefer) |
| **Prometheus server URL** | `https:///metrics-api-bearer` |
| **HTTP method** | `POST` (recommended, handles longer queries) |
| **Scrape interval** | `30s` (matches the upstream Prometheus scrape interval) |
3. Under **Custom HTTP Headers**, click **+ Add header** and set:
* **Header** = `Authorization`
* **Value** = `Bearer cqm_`
4. Click **Save & test**. You should see **"Successfully queried the Prometheus API."**
### Through provisioning (recommended for production)
For reproducible setups, drop this file at `/etc/grafana/provisioning/datasources/cloudquant.yaml`:
```yaml theme={null}
apiVersion: 1
datasources:
- name: cloudquant-liberator
uid: cloudquant-liberator
type: prometheus
access: proxy
url: https:///metrics-api-bearer
isDefault: true
editable: true
jsonData:
httpMethod: POST
timeInterval: 30s
prometheusType: Prometheus
httpHeaderName1: Authorization
secureJsonData:
httpHeaderValue1: "Bearer cqm_"
```
Restart Grafana (or send `SIGHUP`) to pick up the file. The token lives in `secureJsonData`, which Grafana persists encrypted at rest.
## Step 3 — Run a query
Once the datasource is healthy, you can browse it in **Explore** or build dashboards against it. A few queries to verify end-to-end connectivity:
```promql theme={null}
# All scrape targets currently up
sum(up)
# Request rate by HTTP status class through the Liberator gateway
sum by (envoy_response_code_class) (rate(envoy_http_downstream_rq_xx[5m]))
# Liberator waiting-room active connections
liberator_waitingroom_connections_active
# p95 upstream request latency
histogram_quantile(
0.95,
sum by (le) (rate(envoy_cluster_external_upstream_rq_time_bucket[5m]))
)
```
## What's exposed
The Bearer endpoint forwards to the same Prometheus server that powers the in-product **System Monitoring** view, so anything you see there is also queryable here. Notable metric families:
| Prefix | What it covers |
| ----------------- | ---------------------------------------------------------------------------------- |
| `envoy_*` | Envoy Gateway — request rates, latency histograms, upstream connection health |
| `liberator_*` | Liberator application — waiting-room queue, in-flight query times, exporter health |
| `cache_manager_*` | Cache-manager worker pool status and error counters |
| `kube_*` | kube-state-metrics — pods, deployments, nodes, conditions |
| `container_*` | cAdvisor — per-container CPU, memory, filesystem, network |
| `node_*` | node-exporter — host CPU, memory, disk, network |
| `prometheus_*` | The Prometheus server itself — useful for sizing dashboards |
For a complete metric inventory in your environment, query `/api/v1/label/__name__/values` once you've connected.
## Rotating a token
1. Open the **Grafana Integration** dialog as in Step 1.
2. Issue a **new** token for the same logical consumer.
3. Update the new token in Grafana (UI: edit the datasource → swap the `Authorization` header value; provisioning: replace `httpHeaderValue1` and restart).
4. Wait until your Grafana datasource health check passes with the new token.
5. **Revoke** the old token from the same dialog.
This zero-downtime pattern is the same as the [AWS access-key rotation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey) workflow.
## Limitations
* **Read-only.** Prometheus admin endpoints (`/api/v1/admin/tsdb/*`) and the remote-write receiver are not exposed. Even tokened callers cannot mutate the TSDB through this endpoint.
* **One Prometheus.** The endpoint serves the Liberator cluster's primary Prometheus only. Federated views across multiple Liberator clusters require Prometheus-side federation on the consumer side; use `/metrics-api-bearer/federate` with appropriate `match[]` parameters.
* **CORS.** Browser-side direct calls are not supported (the gateway only allows server-side calls). Grafana proxies through its backend, so this restriction does not affect dashboard use.
## Troubleshooting
| Symptom | Diagnosis | Resolution |
| -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Save & test` returns *"non-JSON: `...`"* | The request is being redirected to the OIDC login page — the `Authorization` header isn't being sent, or the token is wrong. | Verify the header name is exactly `Authorization` and the value begins with `Bearer ` (with a trailing space) before the token. |
| `Save & test` returns *401 Unauthorized* | Token has been revoked, never issued, or has a typo. | Re-issue from the **Grafana Integration** dialog and update the datasource. |
| `Save & test` returns *403 Forbidden* | The Liberator gateway rejected the request before reaching Prometheus. | Check with your CloudQuant administrator that the bearer route is enabled on your cluster. |
| Health check passes but every query returns *"no data"* | Time range is outside Prometheus's retention window, or the metric name has changed in a recent upgrade. | Open **Explore**, hit `/api/v1/label/__name__/values` against the datasource, and confirm the metric exists. |
| Range queries fail with *"context deadline exceeded"* | Query window × resolution exceeds the gateway's response timeout. | Use the in-product **System Monitoring** view as a sanity check on the query, then widen your Grafana panel's `interval` or narrow the time range. |
# Integrations Overview
Source: https://knowledge.cloudquant.com/integrations/overview
Connect CloudQuant Data Liberator to AI assistants and observability tools
# Integrations
Connect Liberator to external AI assistants and observability tools. AI clients use the Liberator **MCP** endpoint; Grafana uses the Bearer-authenticated metrics API.
## AI assistants (MCP)
Available in Liberator **2.3** and later. Point your client at the Liberator MCP Endpoint URL from the Liberator UI (for example, `https://api.cloudquant.ai/mcp`), then follow the setup guide for your assistant.
MCP connector with OAuth in ChatGPT Apps
MCP tool with OAuth 2.0 Dynamic discovery
Install the Liberator .mcpb extension bundle
## Observability
Available in Liberator **2.2** and later.
Prometheus metrics via a rotatable Bearer token
## Related
Configure Liberator data source connections and datasets
In-product cluster, queue, dataset, and usage views
# S3 Bucket Setup
Source: https://knowledge.cloudquant.com/integrations/s3-bucket-setup
How to set up your S3 bucket so that CloudQuant Data Liberator can connect and read data from it
# S3 bucket setup for CloudQuant Data Liberator
This guide walks you through setting up your Amazon S3 bucket for use with the CloudQuant Data Liberator platform.
## Prerequisites
* An AWS account with S3 access
* IAM permissions to create buckets and manage policies
* Your CloudQuant account credentials
## Step 1: Create an S3 bucket
1. Navigate to the [AWS S3 Console](https://s3.console.aws.amazon.com/)
2. Select **Create bucket**
3. Select a unique bucket name and region
4. Configure bucket settings as needed
## Step 2: Configure bucket policy
Add a bucket policy that grants CloudQuant Data Liberator read access:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::CLOUDQUANT_ACCOUNT_ID:root"
},
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"
]
}
]
}
```
Contact CloudQuant support for the correct AWS Account ID to use in the bucket policy.
## Step 3: Configure CORS (if needed)
If accessing data from web applications, configure CORS:
```json theme={null}
[
{
"AllowedHeaders": ["*"],
"AllowedMethods": ["GET"],
"AllowedOrigins": ["*"],
"ExposeHeaders": []
}
]
```
## Step 4: Generate access keys
1. Navigate to IAM in the AWS Console
2. Create a new IAM user or use an existing one
3. Generate an Access Key ID and Secret Access Key
4. Provide these credentials to CloudQuant securely
## Step 5: Verify connectivity
Contact CloudQuant support to verify that CloudQuant Data Liberator can successfully connect to your S3 bucket.
## Troubleshooting
| Issue | Solution |
| ---------------- | ---------------------------------------------- |
| Access Denied | Verify bucket policy and IAM permissions |
| Bucket Not Found | Confirm bucket name and region |
| Timeout | Check network connectivity and security groups |
# Introduction
Source: https://knowledge.cloudquant.com/introduction
Welcome to the CloudQuant Documentation - your comprehensive guide to the CloudQuant Data Liberator data platform
# Welcome to CloudQuant
CloudQuant provides institutional-grade financial data infrastructure through the **CloudQuant Data Liberator** platform — a simple point-in-time data access API for live or historical time series data.
## What is CloudQuant Data Liberator?
CloudQuant Data Liberator is CloudQuant's data delivery platform that provides:
* **Multi-language API support** — Python, C#, JavaScript, Java, R, RESTful, C++, and Excel
* **Time series data access** — Query historical and live financial datasets
* **70+ connectors** — so you don't have to move your data to get insights
* **Cloud-native integrations** — S3, Azure Blob Storage, and direct connectivity
## Quick start
Get started with CloudQuant Data Liberator for Python — the most popular way to access data
Explore the full API reference for all supported languages
Browse 70+ available datasets and data integrations
Configure connections to databases, cloud storage, and file systems
## Platform components
| Component | Description |
| --------------------------------- | -------------------------------------------------- |
| **CloudQuant Data Liberator API** | Core data access API with multi-language support |
| **Data Catalog** | 70+ integrated financial and alternative datasets |
| **Integrations** | Cloud platform connectivity (S3, Azure) |
| **Datasource Config** | Connect databases, cloud storage, and file systems |
## Need help?
* Visit the [CloudQuant Support Portal](https://www.cloudquant.com/tickets) for direct assistance
* Browse the documentation sections using the sidebar navigation
# Network Configuration
Source: https://knowledge.cloudquant.com/network-configuration
Configure your network environment for CloudQuant Data Liberator access
# Network configuration
This guide covers the network setup required to connect to the CloudQuant Data Liberator platform.
## Firewall requirements
Ensure the following outbound connections are permitted:
| Protocol | Port | Destination | Purpose |
| -------- | ------- | --------------------------------------------- | ------------------------------- |
| HTTPS | 443 | CloudQuant API endpoints | Data queries and authentication |
| TCP | Various | CloudQuant Data Liberator streaming endpoints | Live data feeds |
## Proxy configuration
If your organization uses a proxy server, configure the CloudQuant Data Liberator client accordingly:
```python Python theme={null}
import os
os.environ['HTTPS_PROXY'] = 'http://your-proxy:8080'
```
```bash Shell theme={null}
export HTTPS_PROXY=http://your-proxy:8080
export NO_PROXY=localhost,127.0.0.1
```
## DNS requirements
Ensure your DNS can resolve CloudQuant's service endpoints. Contact CloudQuant support for the current list of required domains.
## Troubleshooting
If you experience connectivity issues:
1. Verify outbound HTTPS (port 443) is open
2. Check proxy settings if applicable
3. Ensure DNS resolution is working
4. Contact [CloudQuant Support](https://www.cloudquant.com/tickets) for endpoint-specific assistance
# CloudQuant Data Liberator Overview
Source: https://knowledge.cloudquant.com/overview
Understanding the CloudQuant Data Liberator data platform architecture and capabilities
# CloudQuant Data Liberator overview
CloudQuant Data Liberator is CloudQuant's core data delivery platform that provides simple point-in-time data access via API for both live and historical time series data.
## Architecture
CloudQuant Data Liberator serves as the central hub for accessing CloudQuant's data ecosystem:
* **Data Ingestion** — Connects to 70+ data providers and integrates their feeds
* **Data Storage** — Manages time series data with efficient compression and indexing
* **Data Delivery** — Provides API access in multiple programming languages
* **Access Control** — Manages dataset entitlements and user permissions
## Supported languages
CloudQuant Data Liberator provides native client libraries for:
* **Python** — Full-featured client with Pandas DataFrame support
* **C#** — .NET client library
* **JavaScript** — Node.js module
* **Java** — Java client library
* **R** — R language client
* **RESTful** — HTTP REST API for any language
* **C++** — High-performance native client with Apache Arrow support
* **Excel** — Plug-in for direct spreadsheet access
## Key concepts
### Datasets
A dataset is a named collection of time series data. Use `liberator.datasets()` (Python) or equivalent methods to discover available datasets.
### Queries
Queries retrieve data from datasets using parameters like time range (`as_of`, `back_to`), symbols, and column selection.
### Timestamps and MUTS
CloudQuant Data Liberator uses microsecond timestamps (MUTS) for precise time series data alignment.
## Getting started
Choose your preferred language from the [API Reference](/api-reference/concepts/queries-large-datasets) section, or follow the [Python Getting Started Guide](/python-guide/getting-started) for the most common path.
# Understanding as_of and back_to
Source: https://knowledge.cloudquant.com/python-guide/as-of-and-back-to
Learn how the as_of and back_to parameters work with point-in-time datasets in CloudQuant Data Liberator.
# Understanding as\_of and back\_to
The `as_of` and `back_to` parameters define the start and end points for data retrieval. Rather than using "start" and "end," these terms accommodate datasets that are **point-in-time** — meaning different data may be returned based on the `as_of` date.
## What are point-in-time datasets?
Point-in-time datasets reflect data as it existed at a specific historical moment. A common example is revised earnings reports: if your `as_of` date precedes the revision date, you receive original values; if it matches or follows the revision, you get updated figures.
Machine learning datasets increasingly use point-in-time logic. Providers may retrain models and recalculate historical values. You typically want access to the original data from your trading period, and you adopt modified data only when you choose to.
Any point-in-time datasets will be labeled clearly as such in the dataset description in `liberator.datasets()`.
## Source of point-in-time data problems
Standard databases prioritize current information, which introduces **survivorship bias** into quantitative analysis. For instance:
* **SEC filings** — Historical earnings may have been restated due to accounting errors, regulatory scrutiny, or fraud. You need the original reported numbers alongside modification timestamps.
* **Index constituents** — The S\&P 500's composition changes over time. Analysis based on today's index membership creates survivorship bias. For rigorous analysis, you need exact historical index compositions and all changes.
Point-in-time data discipline prevents forward-looking bias that compromises quantitative research integrity.
# Usage of the as_of Parameter
Source: https://knowledge.cloudquant.com/python-guide/as-of-parameter
How to use the as_of parameter to retrieve data as it was known at a particular point in time.
# Usage of the as\_of parameter
The `as_of` parameter allows you to query historical data from a specific moment. You can specify it as a date alone (e.g., `'2024-07-08'`) which CloudQuant Data Liberator interprets as midnight that date, or with specific times down to nanosecond precision.
All CloudQuant Data Liberator times are New York Time (US ET).
## Default behavior
When you omit the `as_of` parameter, CloudQuant Data Liberator assumes the current date and time.
In Liberator **2.4** and later, a future `as_of` on a live-appending dataset is rejected or bounded instead of holding a query worker in an empty-fetch loop. Use `as_of` at or before "now" unless you are intentionally querying forward-dated records that already exist in the source.
## Example query
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-10',
back_to = '2024-07-10',
symbols = ['GOOG', 'AAPL', 'MSFT']))
```
This returns data for the specified date range.
## Without back\_to parameter
If you specify `as_of` but omit or set `back_to` to `None`, the query returns data through yesterday's completed trading bar:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-18',
symbols = ['GOOG', 'AAPL', 'MSFT']))
```
## Jupyter notebook configuration
To display more rows in Jupyter:
```python theme={null}
import pandas as pd
pd.set_option('display.max_rows', 400)
```
Consider your data volume and test with smaller queries initially. See [Query Considerations](/python-guide/query-considerations) for more details.
# Usage of the back_to Parameter
Source: https://knowledge.cloudquant.com/python-guide/back-to-parameter
How to use the back_to parameter to define the start of a time range in CloudQuant Data Liberator queries.
# Usage of the back\_to parameter
The `back_to` parameter is a datetime parameter in CloudQuant Data Liberator queries that defines the start point of a time range for data retrieval.
## Parameter format
You can specify `back_to` in two ways:
**Date only:**
```
'2024-07-08'
```
CloudQuant Data Liberator interprets this as midnight: `'2024-07-08 00:00:00.000000'`
**Specific time:**
You can set it to any time down to nanosecond precision.
## Important notes
All CloudQuant Data Liberator times are New York Time (US ET).
Start with small queries to understand data amounts before requesting large date ranges.
* If `back_to` is omitted or set to `None`, CloudQuant Data Liberator returns the most recent value at the specified `as_of` time.
## Default behavior example
Without `back_to`, the query returns the latest available data:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2023-12-25',
symbols = ['GOOG', 'AAPL', 'MSFT']
))
```
Since December 25, 2023 was not a trading day, CloudQuant Data Liberator returned data from the most recent trading day (December 22, 2023).
# Batch Downloading Data
Source: https://knowledge.cloudquant.com/python-guide/batch-downloading
How to efficiently download large datasets from CloudQuant Data Liberator using batch processing and chunked downloads.
# Batch downloading data
Batch downloading is especially useful when you need to download extremely large datasets. This approach offers several advantages over traditional query methods.
## Key advantages
* **Streamed delivery** — Data arrives as a stream rather than waiting for complete server-side processing
* **Resumable downloads** — Write data to file as it arrives, allowing restart points if connection interrupts
* **Memory efficiency** — Avoids double memory usage common with standard DataFrame downloads
* **Simplicity** — Code length is comparable to standard query-to-DataFrame approaches
## Small batch download example
The simplest implementation submits a single large query and writes batches to file:
```python theme={null}
import liberator, time
start_time = time.time()
for batch in liberator.query(name = 'minute_bars', symbols = None,
as_of = '2024-07-01', back_to = '2024-01-01'):
batch.to_pandas().to_csv("minute_bar_data.csv", mode='a')
print("The query + saving took", (time.time() - start_time)/60.0, " minutes to run")
```
An enhanced version prevents empty files and handles headers correctly:
```python theme={null}
import liberator, time
start_time = time.time()
for i, batch in enumerate(liberator.query(name='daily_bars', as_of = '2024-07-01',
back_to = '2024-01-01', symbols = None)):
if not len(batch):
continue
batch.to_pandas().to_csv('daily_bars.csv', mode='a' if i else 'w',
header=False if i else True)
print("The query + saving took", (time.time() - start_time)/60.0, " minutes to run")
```
## Large batch download process
For very large datasets, splitting downloads into monthly chunks prevents resource constraints:
```python theme={null}
# Define the start and end dates and other setup info for the loop
start_year = 2018
start_month = 3
end_year = 2024
end_month = 06
symbols = None
header = True
individual = 0
header_in_individual = 0
dataset = 'minute_bars'
import liberator, time
back_to_year = start_year
back_to_month = start_month
old_month = None
while (back_to_year < end_year) or (back_to_year == end_year and back_to_month <= end_month):
if back_to_month == 12:
as_of_month = 1
as_of_year = back_to_year + 1
else:
as_of_month = back_to_month + 1
as_of_year = back_to_year
back_to = f"{back_to_year:04d}-{back_to_month:02d}-01"
as_of = f"{as_of_year:04d}-{as_of_month:02d}-01"
print(f"Downloading {dataset} {back_to} to {as_of}", end='')
start_time = time.time()
head = header if (old_month==None) else False
for batch in liberator.query(name=dataset, as_of = as_of,
back_to = back_to, symbols = symbols):
if not len(batch):
continue
if individual:
batch.to_pandas().to_csv(dataset+'_'+back_to+'_'+as_of+'.csv',
mode='a', header=header_in_individual or head)
else:
batch.to_pandas().to_csv(dataset+'_all.csv', mode='a', header=head)
print(" Query + save took", (time.time() - start_time)/60.0, " minutes to run. head:",head)
back_to_year = as_of_year
back_to_month = as_of_month
old_month = as_of_month
```
This monthly chunking approach allows downloading even the largest datasets without memory constraints, making it ideal for production data pipelines.
# Selecting Specific Columns
Source: https://knowledge.cloudquant.com/python-guide/column-selection
How to use the fields parameter to restrict which columns are returned in CloudQuant Data Liberator query results.
# Selecting specific columns
CloudQuant Data Liberator automatically returns a minimum default set of columns for each dataset, which varies but typically includes `_seq`, `muts`, `timestamp`, and `symbol`. Since some datasets contain hundreds of columns, you may want to limit results to specific fields, especially on slower connections.
## Filtering columns with the fields parameter
To reduce returned columns, pass the `fields` parameter with your desired column list:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-22',
symbols = ['AAPL', 'GOOGL'],
fields = ['Open', 'Close']
))
```
CloudQuant Data Liberator places your selected columns at the front of the DataFrame, followed by the default columns.
## Discovering available columns
To identify column names in a dataset, use one of these approaches:
### Using the schema function
```python theme={null}
liberator.datasets(schema=True)['nameOfDataset']
```
### Inspecting a sample query result
```python theme={null}
df.columns
list(df.columns)
print(list(df.columns))
```
Example output for `daily_bars`:
```python theme={null}
['_seq', '_dsname', 'timestamp', 'msg_len', 'msg', 'muts', 'symbol',
'length', 'open', 'high', 'low', 'close', 'volume', 'vwap', 'bvwap',
'spread', 'bidvol', 'askvol', 'count', 'avgdelta', 'Date', 'Time',
'Hour', 'DateTime', 'DateHour']
```
# Checking Dataset Access
Source: https://knowledge.cloudquant.com/python-guide/dataset-access
How to discover which datasets you have access to and view their schemas and descriptions.
# Checking dataset access
CloudQuant Data Liberator provides built-in commands to check your dataset permissions, view available datasets, and access detailed schema information.
## Commands
### Browse the catalog
```python theme={null}
liberator.datasets()
```
Returns datasets visible in the catalog for your account — public datasets and those you are entitled to. Private datasets you are not entitled to are hidden. See [Dataset visibility](/administration/dataset-visibility).
### View entitled datasets only
```python theme={null}
liberator.datasets(entitled=True)
```
Returns datasets you are individually entitled to (in addition to public datasets, depending on client defaults).
### Get dataset details
```python theme={null}
liberator.datasets(details=True)('daily_bars')
```
Retrieves a detailed description of a specific dataset. Replace `'daily_bars'` with your target dataset name.
In Liberator **2.4** and later, details also include catalog-card health: date range covered, symbol count, status (for example, Ready), and last updated. Administrators can configure how often symbol discovery refreshes so new keys appear here after the source changes.
Super Admins can also query **`usage_log`**, a private per-query usage dataset that does not appear in the standard catalog. See [Query usage log](/administration/usage-log).
### View dataset schema
```python theme={null}
liberator.datasets(schema=True)('daily_bars')
```
Returns the complete data dictionary and schema for a specific dataset. Replace `'daily_bars'` as needed.
### Fetch concordance metadata (2.3+)
```python theme={null}
liberator.metadata(name='daily_bars', summary=True, symbol=False)
```
Returns concordance-backed row counts, symbol counts, and start/end timestamps without downloading source rows. See [Dataset metadata](/api-reference/concepts/dataset-metadata) for the `summary` × `symbol` grain matrix.
Use these commands to explore available data before writing queries. The schema information is especially useful for identifying column names to use with the [fields parameter](/python-guide/column-selection).
# Getting Started with CloudQuant Data Liberator for Python
Source: https://knowledge.cloudquant.com/python-guide/getting-started
Learn how to set up and configure the CloudQuant Data Liberator Python library to connect to CloudQuant's data platform.
# Getting started with CloudQuant Data Liberator for Python
This guide introduces the CloudQuant Data Liberator library, available across multiple programming languages via the CloudQuant API platform.
## Basic setup
### Quick start
For environments where CloudQuant Data Liberator is pre-installed, simply use:
```python theme={null}
import liberator
```
### Installation steps
If adding CloudQuant Data Liberator to your own Python environment (local installation, Jupyter, or IDE):
1. Download the required files from your Liberator Profile page
2. Install dependencies:
```bash Conda theme={null}
conda create -n liberator python=3.7
conda activate liberator
conda install -c anaconda requests pyopenssl cryptography six pytz
conda install pandas
conda install -c conda-forge httpx async_generator pyarrow=8.0.0
```
```bash pip theme={null}
python -m venv liberator
source liberator/bin/activate # Windows: liberator\Scripts\activate
pip install requests pandas httpx async_generator "pyarrow==8.0.0" pyOpenSSL cryptography six pytz
```
3. Place files in either:
* The same folder as your code, **or**
* A central Python location (`liberator.py` only — see note below)
Putting `liberator.py` in `site-packages` lets you `import liberator` from anywhere, but `liberator.json` is still read from your working directory unless you set `liberator.auth = '/path/to/liberator.json'`.
To find your Python's site-packages directory:
```python theme={null}
import site
print(site.getsitepackages())
```
### Required files
| File | Purpose |
| ---------------- | -------------------------------------------------------------- |
| `liberator.py` | The Python library |
| `liberator.json` | Pre-populated credentials (username and token) |
| `liberator.pfx` | Security certificate (only required for releases prior to 2.0) |
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx`.
## Server configuration
Files downloaded from your Liberator Profile page include the correct server URL preconfigured in the client.
### Override server URL
To connect to a different instance, set the URL explicitly:
```python theme={null}
liberator.url = 'https://mywidgets.cloudquant.ai/'
liberator.url = 'http://10.10.56.127:47753/'
```
Use `https://` for domain names and `http://` for direct IP addresses. Specify port numbers as shown above when required.
Once your client is configured, you are ready to begin pulling data.
# Accessing Data from a Live Trading Environment
Source: https://knowledge.cloudquant.com/python-guide/live-trading-environment
How to use CloudQuant Data Liberator data in live algorithmic trading environments, including data transfer strategies and live streaming.
# Accessing data from a live trading environment
This guide addresses how to use CloudQuant Data Liberator data in live algorithmic trading environments, particularly when third-party APIs cannot be integrated directly.
## Key challenges
Some organizations restrict third-party APIs in live trading systems. When this applies to you, retrieve the data locally and transfer it to the trading server through approved methods.
## Data transfer solutions
### Low frequency approaches
For infrequently updated data, set up shared file storage accessible to both the live trading application and your local servers. Push data to these locations on a schedule, and coordinate with your Operations department to arrange access.
### High frequency solutions
For more frequent intraday updates, messaging services like [ZeroMQ](https://zeromq.org/) are recommended as lightweight alternatives. ZeroMQ operates peer-to-peer without requiring intermediate broker infrastructure.
## Accessing live data from CloudQuant Data Liberator
CloudQuant Data Liberator delivers live datasets with similar simplicity to historical data retrieval. The platform supports seamless transitions between historical and real-time data.
### Example code
```python theme={null}
import liberator
liberator.url = 'https://weather.cloudquant.ai/'
yesterday = '2024-08-01'
res = liberator.query(name = 'us_live_weather', symbols = '78758', as_of = 'live', back_to = yesterday)
for batch in res:
df = batch.to_pandas()
print(df)
```
When `back_to` is specified, the query delivers historical data from that point forward, then automatically transitions to live streaming. Omitting `back_to` initiates live data immediately at the vendor's defined frequency.
# Pulling Data into a Pandas DataFrame
Source: https://knowledge.cloudquant.com/python-guide/pandas-dataframe
How to extract data from CloudQuant Data Liberator datasets into Python pandas DataFrames with practical examples.
# Pulling data into a Pandas DataFrame
This guide demonstrates how to extract data from CloudQuant Data Liberator datasets into Python pandas DataFrames. The examples use `daily_bars`, a dataset containing US Equity daily OHLCV (Open, High, Low, Close, Volume) data with additional fields.
## Example: pulling daily bars for AAPL
### Multi-line approach
```python theme={null}
today = '2024-06-20'
oneMonthAgo = '2024-05-20'
dataset = 'daily_bars'
mySymbols = 'AAPL'
query = liberator.query(name=dataset, symbols=mySymbols, as_of=today, back_to=oneMonthAgo)
df = liberator.get_dataframe(query)
```
### Single-line approach
```python theme={null}
df = liberator.get_dataframe(liberator.query(name='daily_bars', symbols='AAPL', as_of='2024-06-20', back_to='2024-05-20'))
```
## Key features
* **Flexible symbol input** — accepts single symbols, lists, or `None` (all symbols)
* **Granular time selection** — supports daily, minute, second, and nanosecond resolution
* **Simple syntax** — collapsible into single-line queries for efficiency
The query returns a properly formatted pandas DataFrame containing the requested historical data.
# Considerations When Submitting a Query
Source: https://knowledge.cloudquant.com/python-guide/query-considerations
Important constraints, timestamp conventions, and best practices to understand before querying CloudQuant Data Liberator datasets.
# Considerations when submitting a query
Before submitting your first CloudQuant Data Liberator query, there are several important factors to keep in mind regarding constraints, timestamps, and data volume.
## Constraints on a query
The system imposes only two limitations: the **time range** selected and the **symbols** chosen for analysis.
All timestamps in CloudQuant Data Liberator use **New York Time (US ET)**, regardless of the market's geographic location. This standardization helps you understand when data becomes available to you.
## Notes on timestamps
Datasets frequently contain multiple timestamps serving different purposes:
* The **primary timestamp** indicates when you can access the data.
* **Secondary timestamps** may represent exchange or execution times, which typically occur before or coincide with the main timestamp.
For calculated datasets like minute bars, there is often a `bar_time` distinct from the availability timestamp. In US equities markets, the first daily bar concludes at `9:30:59.999999`, with a `bar_time` of `9:31:00.000000`.
Calculated datasets may show vendor timestamps differing from CloudQuant's timestamp. CloudQuant's timestamp is designed to give you the best possible idea of the **actual available time of the data**.
## Think before you query
Before submitting queries, assess the likely data volume.
A simple request for AAPL daily bars for April might return approximately 30 rows. The same parameters applied to NBBO data (all bids and asks) could generate enormous datasets requiring extensive system processing.
Best practice involves querying unfamiliar datasets with **limited date ranges initially**, then progressively expanding timeframes as you become comfortable with the data scale.
In Liberator **2.4** and later, administrators can also set a **maximum number of days** a single query may span on a dataset. If you hit that cap, the error message names the limit and the configured value — narrow `back_to` / `as_of` and retry. See [Fine-grained permissions](/administration/fine-grained-permissions#limits-and-settings).
For guidance, contact [customer\_success@cloudquant.com](mailto:customer_success@cloudquant.com).
# Query Parameter Combinations
Source: https://knowledge.cloudquant.com/python-guide/query-parameters
A summary of all CloudQuant Data Liberator query parameters and how different combinations affect the results you receive.
# Query parameter combinations
This guide explains the main parameters for CloudQuant Data Liberator queries and the expected results from different combinations.
## Core parameters
| Parameter | Description | Default |
| --------- | ------------------------------------------------------------------------------------------ | -------------------- |
| `name` | Dataset identifier (required) | -- |
| `symbols` | Single symbol, list, or (2.4+) `LIKE` pattern with `%`, e.g. `['AAPL','GOOG']` or `'PWR%'` | `None` (all symbols) |
| `back_to` | Query start date | `None` |
| `as_of` | Query end date | Current date/time |
| `fields` | Specify desired columns to reduce download time | All columns |
| `stats` | Set to `'Total'` for symbol counts instead of full results | -- |
| `where` | Custom SQL predicate appended to the generated filters | `None` |
| `sql` | Full raw SQL query (replaces `name`, time range, and other construction params) | `None` |
Use `liberator.datasets(entitled=True)` to list all datasets available to you.
See the [Query Parameters Reference](/api-reference/concepts/query-parameters) for the full parameter list, including superquery, N-Query, and authentication options.
## Query examples
### All parameters included
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-16',
symbols = ['AAPL', 'GOOGL']))
```
Returns data for specified symbols within the date range.
### With a symbol pattern (2.4+)
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'energy_contracts',
as_of = '2026-08-02',
back_to = '2026-08-01',
symbols = 'PWR%'))
```
An unescaped `%` is a `LIKE` wildcard, so this returns every key that starts with `PWR`. Mix exact keys and patterns in one list (`['AAPL', 'PWR%']`). Escape a literal percent as `\%`. See [Pattern filters](/python-guide/symbols-and-key-fields#pattern-filters-24).
### Without symbols
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-16'))
```
Defaults to all available symbols.
### Without `as_of`
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
back_to = '2024-07-22'))
```
Defaults `as_of` to current time, returning data from present back to specified date.
### Without `back_to`
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2023-12-25'))
```
Returns the most recent data point per symbol as of the specified date. Look-back distance varies by dataset density.
### Minimal query (name only)
```python theme={null}
df = liberator.get_dataframe(liberator.query(name = 'daily_bars'))
```
Returns the latest data point for all symbols in the dataset.
## Custom `where` filters
Use `where` with a standard parameter-based query to append extra SQL predicates on top of the filters CloudQuant Data Liberator generates for time range and symbols. The clause is combined with `AND` — do not include the `WHERE` keyword.
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name='daily_bars',
as_of='2024-07-24',
back_to='2024-07-16',
symbols=['AAPL', 'GOOGL'],
where='volume > 1000000'
))
```
`where` works with time-range, LKV, and N-Query modes. Column names that match the dataset schema are automatically quoted when needed.
### Multi-dataset / concordance-style filters
For datasets keyed by more than symbol (for example concordance tables with a `dataset` column), use `where` to narrow the result set:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name='concordance',
as_of='2025-01-01',
back_to='2024-01-01',
where="dataset = 'my_derived_dataset'"
))
```
`where` cannot be combined with `sql`. Use one approach or the other.
## Raw `sql` queries
Pass a complete SQL `SELECT` statement via `sql` instead of `name`, `back_to`, `as_of`, and related parameters. When `sql` is present, parameter-based query construction is skipped.
```python theme={null}
df = liberator.get_dataframe(liberator.query(
sql='''
SELECT *
FROM "daily_bars"
WHERE muts >= 1704067200000000
AND muts < 1704153600000000
AND symbol = $$AAPL$$
''',
user='my_user',
system='API'
))
```
### SQL requirements
* The statement must be a **`SELECT`** (including `UNION` / `INTERSECT` / `EXCEPT` branches).
* Every `SELECT` arm must include an explicit **`WHERE`** clause.
* By default, the `WHERE` clause must include a **time-range predicate** on `muts` or another configured column (`=`, `>`, `>=`, `<`, `<=`, or `BETWEEN`). Use microsecond timestamps for `muts` values.
* Quote dataset names with double quotes when they contain special characters.
* Use `$$symbol$$` dollar-quoting for string literals in filters.
### Parameters allowed with `sql`
When using `sql`, only connection, auth, and transfer options are permitted alongside the statement — for example `user`, `system`, `token`, `compress`, `json_xfer`, and `batch_size`. Supplying non-empty values for `name`, `back_to`, `as_of`, `where`, or other construction parameters raises an error.
Raw SQL bypasses the convenience of `name` / `symbols` / `back_to` / `as_of`. Prefer parameter-based queries unless you need full control over the generated SQL.
# Required Timestamps in Queries
Source: https://knowledge.cloudquant.com/python-guide/required-timestamps
Every Liberator query must include a valid date or time range using back_to and as_of (or an explicit WHERE on time columns)
# Required timestamps in queries
Liberator enforces a **time-based predicate on all queries**. Every query must include a valid, non-zero date or time range. Queries without one return an error and no data.
Previously, a query with no time parameters could return an entire dataset. That behavior was removed to prevent unexpectedly large result sets and unnecessary load.
Review saved notebooks, scripts, and scheduled jobs. Any query that omits `back_to` / `as_of` (or an explicit time `WHERE` in raw `sql`) will fail until updated.
Applies to Liberator **2.1 and later**.
## Valid timestamps
A valid time scope must:
* Cover a **specific date, date range, or datetime range** — not zero, null, or empty.
* Use **`back_to` and `as_of`** for parameter-based queries (Liberator's point-in-time window). See [Understanding as\_of and back\_to](/python-guide/as-of-and-back-to).
* Fall within the dataset's available coverage (valid dates outside coverage return an **empty** result, not a timestamp error).
### Correct examples
```python theme={null}
import liberator
# Single day
df = liberator.get_dataframe(liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2026-01-01",
as_of="2026-01-01",
))
# Date range
df = liberator.get_dataframe(liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2026-01-01",
as_of="2026-06-30",
))
# Datetime range
df = liberator.get_dataframe(liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2026-06-01 09:30:00",
as_of="2026-06-01 16:00:00",
))
```
### Incorrect examples
```python theme={null}
# Missing back_to and as_of — error
liberator.query(name="my_dataset", symbols=["AAPL"])
# Null dates — error
liberator.query(name="my_dataset", symbols=["AAPL"], back_to=None, as_of=None)
```
### Raw `sql` queries
Every `SELECT` must include an explicit **`WHERE`** clause with a time-range predicate on `muts` or another configured time column. See [Query parameters — Raw SQL](/python-guide/query-parameters#raw-sql-queries).
## What the error looks like
The API or client returns an error indicating a time-based predicate is required. This is expected enforcement, not a platform failure.
## Update existing queries
1. **Find affected calls** — Search for `liberator.query()` without `back_to` / `as_of`, or raw `sql` without a time `WHERE`.
2. **Choose the right window** — Daily jobs: parameterize to yesterday or today. Historical pulls: set explicit start and end dates.
3. **Add parameters** — Use the examples above.
4. **Test** — Run manually before re-enabling schedules.
## FAQ
**I used to pull a full dataset for reference. What now?**\
Set `back_to` and `as_of` to the dataset's available range (ask your administrator or check dataset details in the admin portal).
**Error but I provided dates?**\
Confirm values are not null/empty, `back_to` is not after `as_of`, and dates use `YYYY-MM-DD` or `YYYY-MM-DD HH:MM:SS`.
**Valid dates but empty results?**\
The timestamp is accepted; the range may fall outside dataset coverage.
**AI assistant queries?**\
Assistants must specify a time range. Timeless requests should prompt for dates; otherwise the query fails like any other.
## Related
Point-in-time semantics for Liberator datasets
Full parameter reference including raw SQL rules
# Symbols and Key Fields
Source: https://knowledge.cloudquant.com/python-guide/symbols-and-key-fields
Understanding the symbol parameter in CloudQuant Data Liberator queries, how to discover available symbols, and how to query for all symbols in a dataset.
# Symbols and key fields
The **symbol** parameter serves as the key for querying datasets in CloudQuant Data Liberator. The platform initially focused on US Equity Symbols, but its datasets have expanded to include diverse data types such as shipping, weather, and market research. These datasets use varied keys like zip codes and geographic locations. CloudQuant Data Liberator keeps the `symbols` parameter name for consistency with existing integrations.
**Default Value:** `None` (returns all symbols/keys, including blanks/NaN values)
`symbols` and `keys` are synonyms — use either name.
When setting your query parameters, consider the amount of data you may be requesting and start small until you understand data volume patterns for each symbol/date combination.
## Determining unique symbols in a dataset
The most straightforward approach is to query and count the data. Here is an example using the `daily_bars` dataset:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-22',
symbols = None))
df
```
This returns over 20,000 rows. To count unique symbols:
```python theme={null}
len(df.symbol.unique())
# Output: 10524
```
## Storing and displaying symbols
To store symbols in a variable:
```python theme={null}
allsym = list(df.symbol.unique())
```
To display the complete list (with caution — lists can be lengthy):
```python theme={null}
print(list(df.symbol.unique()))
```
## Query methods for all symbols
Either explicitly set `symbols = None` or omit the parameter entirely, as it defaults to `None`.
## Pattern filters (2.4+)
In Liberator **2.4** and later, `symbols` / `keys` accept SQL `LIKE` patterns. An unescaped `%` matches any sequence of characters, so you can request a product family or naming pattern without listing every contract.
```python theme={null}
# All keys that start with PWR
df = liberator.get_dataframe(liberator.query(
name='energy_contracts',
symbols='PWR%',
back_to='2026-08-01',
as_of='2026-08-02',
))
# Mix exact keys and patterns
df = liberator.get_dataframe(liberator.query(
name='energy_contracts',
symbols=['AAPL', 'MSFT', 'PWR%'],
back_to='2026-08-01',
as_of='2026-08-02',
))
```
| Pattern | Matches |
| ------- | --------------------------------------------- |
| `PWR%` | `PWR`, `PWR-H`, `PWR_CAL_26` |
| `G%GL` | `GOGL`, `GAGL`, `GOOGLE` |
| `A\%PL` | only the literal key `A%PL` (escaped percent) |
Exact keys (no unescaped `%`) still use equality, including lists of tickers. Liberator only switches to `LIKE` for values that contain `%`.
Start with a short date range when you first use a pattern. A leading `%` on a large dataset can return far more rows than an exact-symbol query.
See the [query parameters reference](/api-reference/concepts/query-parameters#dataset--symbol-parameters) for `symbols` and `keys`.
# Security & Access
Source: https://knowledge.cloudquant.com/security-and-access
CloudQuant Data Liberator security model, authentication, and access control
# CloudQuant Data Liberator security and access
CloudQuant Data Liberator provides enterprise-grade security for data access and distribution.
## Authentication
Access to CloudQuant Data Liberator requires valid credentials provided by CloudQuant. Authentication is handled through:
* **API Keys** — Used for programmatic access via client libraries
* **User Credentials** — Used for interactive access via Excel and CQAI
## Dataset entitlements
Access to specific datasets is controlled through entitlements:
* Each user or organization is granted access to specific datasets
* Use `liberator.datasets()` (Python) or equivalent to see your available datasets
* Organization administrators can manage users and entitlements from the admin portal — see [Administration Overview](/administration/overview)
* Contact CloudQuant support to request access to additional datasets
### For administrators
Add, create, deactivate users and grant or revoke dataset access
Restrict query date ranges, row limits, keys, and columns
## Network security
CloudQuant Data Liberator supports secure connectivity through:
* **TLS/SSL encryption** for all data in transit
* **IP whitelisting** for restricted access environments
* **VPN connectivity** for on-premises deployments
## Anti-virus considerations
When running CloudQuant Data Liberator client libraries locally, ensure your anti-virus software does not block:
* CloudQuant Data Liberator client executables
* Network connections to CloudQuant data endpoints
* Local cache directories used by the client
See the [Network Configuration](/network-configuration) page for detailed setup instructions.
# System Monitoring Overview
Source: https://knowledge.cloudquant.com/system-monitoring/overview
Built-in observability for the CloudQuant Data Liberator stack, with a tour of each tab and when to look at it
# System monitoring
The **System Monitoring** page in the Liberator UI surfaces the most operationally relevant signals from the Liberator stack in a single place: cluster health, queue depth, dataset access patterns, long-running queries, and license usage.
It's intended for super-admins, on-call engineers, and capacity planners. End users do not see this page.
The System Monitoring page is backed by the same Prometheus instance you can connect Grafana to. See the [Grafana](/integrations/grafana) integration guide if you want the same data alongside metrics from systems outside CloudQuant.
## Opening system monitoring
1. Sign in to the Liberator UI as a super-admin.
2. Click **System Monitoring** in the top navigation.
If the menu item doesn't appear, your account doesn't have super-admin privileges. Contact your CloudQuant administrator.
## Tabs
System Monitoring is organized into five tabs. The first four are backed by Prometheus and share a global **time-range selector** (`6h` / `24h`) in the upper-right. The fifth is backed by the entitlements database and uses its own dedicated `1d / 1w / 1m / 1y` selector.
### Cluster
Real-time cluster health from the Liberator gateway, application pods, and host nodes.
What you'll see:
* Gateway request rate and latency percentiles (`p50` / `p95` / `p99`)
* Per-pod CPU and memory utilization for Liberator components
* Per-node CPU, memory, and filesystem utilization
* Data-cache worker pool status
Watch **volume / disk usage** on mounted filesystems. Sustained usage at or above roughly **85%** on a volume warrants immediate attention to avoid query failures from insufficient write space.
Use this tab to answer: *"Is the cluster behaving normally right now, and if not, where is the problem?"*
### Queue
The Liberator **waiting room** queue — how many requests are queued, how long they've been waiting, and which users own them.
What you'll see:
* Active connections (in flight) and queued connections (waiting)
* Per-user breakdown of queue occupancy
* Maximum in-flight query duration (a useful early-warning signal)
The `CQAIOps` service account often appears as a high-volume user; that reflects automated platform monitoring and is expected.
Use this tab to answer: *"Is anyone being blocked, and by whom?"*
### Datasets
Dataset access patterns over the selected time range.
What you'll see:
* Top datasets by query count
* Top datasets by bytes returned
* Distribution of access by client (Python, REST, Excel, etc.)
Use this tab for capacity planning and detecting anomalous access patterns (e.g. a previously dormant dataset suddenly receiving heavy traffic).
### Long queries
The slowest individual queries in the selected window.
What you'll see:
* A ranked list of queries with execution time, user, dataset, and `from` / `to` window
* Click-through to see the full query text and result-set size
Use this tab to find candidates for query rewriting, dataset re-partitioning, or user education.
### Usage
License utilization from the entitlements database. Distinct from the other tabs in two ways:
1. **Different selector.** This tab exposes `1d / 1w / 1m / 1y` windows instead of the Prometheus `6h / 24h`, because license utilization is measured against per-contract caps that operate on much longer windows.
2. **Different backing store.** Numbers come from the entitlements database, not Prometheus, so they survive Prometheus retention rollovers and reflect contract truth.
What you'll see:
* Active vs. licensed seat count, by license tier
* Per-dataset utilization vs. contract caps
* Trend lines that make it easy to spot accounts approaching their limits
* **Most queried datasets** — use this to prioritize [cache pre-generation](/administration/cache-pre-generation)
The `1d` / `1w` selector labels on this tab may still reflect monthly aggregation in the backing entitlements store in some releases. Treat long-window utilization as directional until label semantics match the aggregation period in your environment.
For **per-query** records (who queried which dataset, with which `as_of` / `back_to`, rows returned, and outcome), Super Admins can query the `usage_log` dataset. See [Query usage log](/administration/usage-log).
## Grafana integration
The action in the upper-right of every tab opens the **Grafana Integration** dialog. Super-admins can use it to:
* **Issue Bearer tokens** for external Prometheus consumers (the full token is shown exactly once at creation, so copy it immediately).
* **List existing tokens** with their issue time and issuing user.
* **Revoke tokens** that are no longer needed or may have leaked.
See the full setup walkthrough in the [Grafana integration guide](/integrations/grafana).
## How the data flows
```
┌────────────────────────────────────────────────────────────────────┐
│ Liberator UI │
│ ┌────────────────────────┐ ┌──────────────────────────────┐ │
│ │ Cluster / Queue / │ │ Usage tab │ │
│ │ Datasets / Long Q │ │ │ │
│ └──┬─────────────────────┘ └──┬───────────────────────────┘ │
└─────┼────────────────────────────┼────────────────────────────────┘
│ /metrics-api/* (OIDC cookie)│ /admin-api/entitlements/*
▼ ▼
┌──────────────┐ ┌──────────────────┐
│ Prometheus │ │ Entitlements DB │
│ (read-only) │ │ (PostgreSQL) │
└──────────────┘ └──────────────────┘
▲
│ /metrics-api-bearer/* (Bearer token)
│
┌──────────────┐
│ External │
│ Grafana, │
│ Federation │
└──────────────┘
```
Both Prometheus-fronted endpoints (`/metrics-api/*` for the in-product UI and `/metrics-api-bearer/*` for external consumers) expose the same read-only subset of the Prometheus HTTP API. The OIDC-fronted route is what the in-product tabs use; the Bearer-fronted route is what Grafana and federated Prometheus servers use.
# What's New in Liberator 2.1
Source: https://knowledge.cloudquant.com/whats-new/liberator-2.1
Release notes for the CloudQuant Data Liberator 2.1 release — enterprise SSO, concurrent query limits, broader file and date-format support
# What's new in Liberator 2.1
Liberator 2.1 brings two large workstreams together into a single release: **enterprise SSO** for the Liberator UI, and **concurrent-query governance** so a single user can no longer monopolize platform capacity.
It also picks up a long tail of dataset-quality improvements — broader date-format coverage, a new file type, better handling of records containing special characters, and **[required timestamps on all queries](/python-guide/required-timestamps)**.
## Highlights
The Liberator UI now supports standards-based single sign-on (SAML/OIDC), including federation with your existing identity provider and a customizable login page.
Per-user concurrent-query caps prevent runaway clients from starving capacity.
Liberator now ingests pipe-separated value (`.psv`) files alongside CSV and Parquet, with the same date-format detection improvements applied to all file types.
## Enterprise SSO
Liberator 2.1 promotes a standards-based **single sign-on** flow to the default authentication path for the Liberator UI:
* **Federation with your identity provider** — your users sign into Liberator using their existing corporate SSO session. There's nothing to configure on the user's side, since CloudQuant wires the federation up for your tenant.
* **Customizable login page** — the login page can be styled to match your organization's branding rather than the default look.
* **Logout reliability** — fixed a bug where clicking *Log out* would leave a stale session in some cases, causing the user to be silently signed back in on the next page navigation. The sign-out flow now consistently clears the session on both the gateway and the identity provider.
## Concurrent query limits
A single user issuing dozens of long-running queries can no longer monopolize platform capacity. Liberator now enforces a per-user concurrent-query cap; requests above the cap are queued in the waiting room and processed FIFO. The cap is visible to administrators via the **System Monitoring → Queue** tab introduced in 2.2.
## New file / data-format support
* **PSV (pipe-separated value) files** — Liberator now treats `.psv` as a first-class file type alongside `.csv` and `.parquet`. The auto-detection logic mirrors CSV: header row, type inference, configurable null sentinel.
* **Additional date-format support** — Parquet file uploads with date columns that previously failed during the date-processing step (mixed `YYYY-MM-DD` / `MM/DD/YYYY` / Unix-epoch columns in the same file) now ingest cleanly. The detection is per-column, not per-file.
* **Special-character handling in records** — records containing characters that previously needed manual escaping (multi-line strings, embedded quotes, non-ASCII symbols) round-trip through the API unchanged.
## Performance & reliability
* **Faster small-query response times** — reused connections between Liberator's query and cache tiers shave latency off small queries and reduce pressure on the cache tier under heavy concurrent load.
* **Better diagnostics on cache miss** — when the platform misses cache on a symbol lookup it now records the requested symbol set, making "why is this symbol always being recomputed?" answerable from logs alone.
## Bug fixes
* **Options dataset onboarding** — a previously unsupported options intraday-history dataset is now onboardable through the standard ingestion path.
* **Pivot on uploaded files** — pivot operations on Liberator UI–uploaded files that worked in earlier releases but stopped after a recent cut are working again.
* **Dataset descriptions show display names** — the Liberator UI's dataset description card now renders the dataset's friendly display name instead of falling back to the internal slug.
* **S3-backed dataset date parsing** — fixed a date-parse path that mis-handled certain S3-backed datasets with mixed date-column formats.
## Subsequent fixes shipped in 2.1.1
Some issues that surfaced after 2.1 were fixed in the [2.1.1 patch](/whats-new/liberator-2.1.1), most notably a reliability issue under sustained SQL-source load.
# What's New in Liberator 2.1.1
Source: https://knowledge.cloudquant.com/whats-new/liberator-2.1.1
Patch-release notes for CloudQuant Data Liberator 2.1.1 — reliability under heavy SQL-source load, configurable branding, improved error messages, PCAP FIX delegate
# What's new in Liberator 2.1.1
Liberator 2.1.1 is a **patch release** on top of [2.1](/whats-new/liberator-2.1).
The headline change is a reliability fix that surfaced under sustained SQL data-source load. The remainder of the release is customer-driven polish: per-organization branding controls, friendlier error messages, and a new **PCAP FIX delegate** for low-latency tick consumers.
## Highlights
A blocking call in the data-cache tier could stall the platform under sustained SQL-source ingestion. Fix moves the call off the request-handling path.
The Liberator UI nav-bar logo and login-page logo can now be customized per organization, so your own branding appears in front of your users.
New delegate path for consuming FIX-encoded tick data captured from PCAP, alongside the existing Liberator delegate types.
Liberator error responses now surface root-cause hints rather than a raw exception chain.
## Reliability improvements
* **Stall under sustained SQL-source load** — under sustained ingestion against a SQL-backed datasource, a blocking call in the data-cache tier could occupy the request-handling path and cause the platform to go unresponsive for tens of seconds at a time. The fix moves the blocking call onto a background worker so request handling stays responsive while the SQL backend is busy.
* **Mapping-data fetch resilience** — fixed a regression where mapping-data fetches could error with *"No mapping data returned for ..."* even when the upstream data was present. The fetch retries now distinguish between transient transport errors (which retry) and structurally empty responses (which don't).
* **Force-regen no longer silently returns 0 rows** — the Python client's `force_regen=True` path could return 0 rows when an underlying data-cache regeneration failed silently. Regenerations now surface failures as exceptions rather than returning an empty result.
## Configurable branding
Per-organization branding is now supported across the Liberator UI:
* **Login-page logo**
* **Liberator UI nav-bar logo**
Both default to the CloudQuant logo unless your organization has requested a custom configuration. Contact your CloudQuant account team to set up custom branding for your tenant.
## SSO for the Liberator UI
The enterprise SSO integration introduced in 2.1 is now wired into the **Liberator UI** login flow as well. Federated users see a single "Sign in" button on the login page. Clicking it bypasses the username/password prompt and authenticates via the corporate SSO session if one is already active in the browser.
## PCAP FIX data delegate
Liberator now ships a **PCAP FIX delegate**, a new delegate type that ingests FIX-protocol messages from PCAP captures, decoded and presented to the standard Liberator query surface. This is intended for low-latency tick consumers that capture exchange feeds at the wire level and want to query them through the same API as their historical datasets.
## Improved error messages
Liberator's error responses across the Python client, REST API, and Liberator UI have been re-paved so that:
* The *root* cause is reported first, with the call-site trace as supplementary context (previously the call-site trace was first, burying the actual cause).
* Common configuration mistakes (missing entitlement, malformed date parameter, unknown dataset) get plain-English explanations instead of a raw exception class name.
* Network-transport errors are distinguishable from server-side errors in the response shape, so client libraries can apply different retry behavior to each.
# What's New in Liberator 2.2
Source: https://knowledge.cloudquant.com/whats-new/liberator-2.2
Release notes for the CloudQuant Data Liberator 2.2 release — System Monitoring, Grafana integration, scheduled cache creation, ETL on-the-fly mapping UI, and reliability improvements
# What's new in Liberator 2.2
Liberator 2.2 focuses on **observability and operability**. The release adds an in-product System Monitoring page for super-admins and a standardized way to plug external Grafana instances into the underlying metrics. It also introduces **scheduled cache creation**, which removes cold-start latency for predictable access patterns, and the foundations of an **ETL on-the-fly dataset mapping** workflow.
It also lands a reliability fix that prevents long-running query workers from getting stranded across platform restarts.
## Highlights
A new admin-only page surfacing cluster health, queue depth, dataset access patterns, long-running queries, and license utilization.
A Bearer-authenticated, read-only metrics endpoint at `/metrics-api-bearer/`, with token issuance and revocation built into the Liberator UI.
Pre-generate dataset caches on a schedule so users never hit a cold-cache penalty for predictable access patterns.
Define cross-dataset value correspondences and enrich queries with `translate=true`.
Query workers no longer get stranded on stale connections when the platform restarts behind them.
System Monitoring's Usage tab uses its own license-appropriate time window, and tab state is preserved as users navigate.
## System monitoring
A new super-admin-only **System Monitoring** page consolidates the operational signals previously scattered across logs and ad-hoc queries. Five tabs:
| Tab | Time-range selector | What it answers |
| ---------------- | ------------------- | -------------------------------------------------------------- |
| **Cluster** | `6h / 24h` | Is the cluster behaving normally right now? |
| **Queue** | `6h / 24h` | Is anyone being blocked, and by whom? |
| **Datasets** | `6h / 24h` | What are users actually querying? |
| **Long Queries** | `6h / 24h` | Which individual queries are slow enough to warrant attention? |
| **Usage** | `1d / 1w / 1m / 1y` | How close are we to our license caps? |
See the [System Monitoring overview](/system-monitoring/overview) for a walkthrough.
## Grafana integration
CloudQuant Data Liberator now exposes a standardized, Bearer-authenticated metrics endpoint at `https:///metrics-api-bearer/`. The endpoint speaks the full standard Prometheus HTTP API surface (`/api/v1/query`, `/api/v1/query_range`, `/api/v1/series`, `/api/v1/labels`, `/api/v1/label//values`, `/api/v1/status/buildinfo`, `/federate`). Any off-the-shelf Prometheus client — Grafana, Mimir, federated Prometheus, custom scripts — can consume it without modification.
Token lifecycle is fully self-service from the Liberator UI:
1. **Issue** — a super-admin clicks **Generate token** in the **Grafana Integration** dialog. The full token value is shown exactly once.
2. **Use** — point Grafana at `/metrics-api-bearer` with `Authorization: Bearer cqm_` as a custom HTTP header.
3. **Rotate** — issue a new token, switch Grafana to it, verify, then revoke the old one (zero-downtime).
4. **Revoke** — same dialog, list view.
See the full walkthrough at [Grafana integration](/integrations/grafana).
## Scheduled cache creation
For datasets with predictable access patterns (daily morning reports, weekly market-open snapshots), Liberator now supports **scheduled cache creation**. Super-admins set a cron-style schedule per dataset directly in the Liberator UI's caching settings. The platform pre-generates the cache slice before users query it, so the first read of the day hits a warm cache rather than triggering a generation.
See the [cache pre-generation guide](/administration/cache-pre-generation) for the full Super Admin workflow.
## ETL mapping UI
Liberator 2.2 lays the foundation for the **ETL on-the-fly** workflow. Analysts define cross-dataset value correspondences (e.g. "the `ticker` column in Dataset A maps to the `symbol` column in Dataset B, with these reconciliations"). Liberator then surfaces a *virtual unified column* across both datasets at query time, without a heavyweight pre-materialization step.
What lands in 2.2:
* **ETL mapping UI design** — UX research and component specs.
* **Data Translation Mapping** — the underlying admin-API surface for storing and retrieving mapping definitions.
* **Dataset Field Mapping UI** — the in-Liberator-UI workflow for defining correspondences and previewing the virtual unified column. See [Dataset field mapping](/administration/dataset-field-mapping).
Full GA of the mapping workflow continues in later releases; see [What's New in Liberator 2.3](/whats-new/liberator-2.3) for related AI and connectivity work shipped after 2.2.
## Reliability improvements
A user-facing reliability bug is fixed in 2.2: previously, after a behind-the-scenes platform restart, a small number of long-running queries could hang indefinitely rather than reconnecting. In 2.2 those workers detect the broken connection within a short keepalive window and reconnect automatically, so long-running queries continue cleanly across platform restarts instead of stalling.
## UI changes
* **Usage tab time-range selector.** The global `6h / 24h` selector is now hidden when the **Usage** tab is active. Usage has its own `1d / 1w / 1m / 1y` window because license utilization is measured against per-contract caps that don't make sense on a 6-hour window.
* **Controlled Tabs.** The System Monitoring page's tab component is now fully controlled, so per-tab state survives switches without remounting, and the token-rotation flow triggers a clean cache reset across all tabs.
## Also in 2.2
Smaller items in the same release:
* **OneDrive URL share-link paste** — paste a full OneDrive or SharePoint share URL into a connection form; see [SharePoint and OneDrive](/datasource-config/sharepoint-onedrive).
* **Email verification** — new accounts receive an email verification message when provisioned.
* **Connection-form example text** — UX hints in connection-source forms so users know what shape of input each field expects (host, bucket name, region, etc.) before they make a mistake.
## Compatibility
There are no breaking changes to the client libraries (Python, REST, JavaScript, Excel, etc.) or to the dataset query surface, so existing code and queries continue to work unchanged. The new features in 2.2 are surfaced through the Liberator UI and the Bearer-token metrics endpoint.
To start using the Grafana integration, a super-admin generates a Bearer token from the **Grafana Integration** tab in the Liberator UI. See the [Grafana integration](/integrations/grafana) guide for the full walkthrough.
# What's New
Source: https://knowledge.cloudquant.com/whats-new/overview
Release notes and change history for the CloudQuant Data Liberator platform
# What's new
Release notes for the CloudQuant Data Liberator platform. Each version page summarizes the user-visible changes in that release.
**Latest.** Symbol pattern filters, per-query `usage_log`, clearer entitlement errors, query day-range caps, dataset health cards, Excel plugin in the UI, and more reliable connections.
AI-driven dataset onboarding, AI-driven reporting, HubSpot connectivity, invite-via-email onboarding, SharePoint/OneDrive wildcards, and reliability improvements.
In-product System Monitoring, Grafana integration, scheduled cache creation, ETL on-the-fly mapping UI, and reliability improvements.
Stability patch. Reliability improvements under heavy SQL-source load, configurable branding, friendlier error messages, and a new PCAP FIX delegate.
Major release. Enterprise SSO, concurrent query limits, PSV file support, and broader date-format coverage.
## Versioning
Liberator uses **semantic-style versioning** (`MAJOR.MINOR.PATCH`):
* **MAJOR / MINOR** versions (e.g. `2.3` → `2.4`) bundle new features alongside fixes.
* **PATCH** versions (e.g. `2.1` → `2.1.1`) are restricted to fixes and small enhancements.
CloudQuant manages the rollout of new versions to your environment. Reach out to your CloudQuant account team if you have questions about when a specific change reaches you.
# Fine-Grained Permissions
Source: https://knowledge.cloudquant.com/administration/fine-grained-permissions
Restrict user query scope with trial periods, date ranges, row limits, keys, and column filters
# Fine-grained permissions
Fine-grained permissions let Super Admins apply additional restrictions on top of standard dataset entitlements. Use them to enforce trial periods, limit historical lookback, cap query volume, or restrict which keys and columns a user can access.
Fine-grained rules apply **per dataset** or **per dataset group** and are configured from the user's **Data Permissions** dialog.
## When to use fine-grained permissions
| Scenario | Recommended restriction |
| ----------------------------------- | --------------------------------------- |
| Time-limited trial | **Trial Range** |
| Historical data only (no live feed) | **Data Query Range** → Historical Only |
| Rolling window (e.g., last 90 days) | **Data Query Range** → Rolling Window |
| Fixed evaluation period | **Data Query Range** → Fixed Date Range |
| Cap the width of a time-range query | **Max Query Days** |
| Cap download size | **Max Rows** |
| Limit API usage | **Max Queries** |
| Restrict to a symbol universe | **Keys** |
| Hide sensitive columns | **Columns** |
Fine-grained permissions require an underlying dataset or group entitlement. They cannot grant access on their own — the user must already be entitled to the dataset or group.
## Opening fine-grained settings
From the Users page, click the data-permissions icon for the target user.
Ensure the dataset or dataset group row is checked in the permissions table.
Click the gear icon (**Actions** column) on the selected row. The icon appears filled (blue) when fine-grained rules already exist for that item.
Enable the restriction sections you need, set values, click **Done** to return to the permissions table, then **Save Changes**.
Fine-grained settings are not saved until you click **Save Changes** on the main Data Permissions dialog. Clicking **Done** only returns you to the entitlement table.
## Restriction sections
Each section is controlled by an **Enable** toggle. Disabled sections are ignored — their fields are not sent to the backend.
### Trial Range
Limits when the user can access the platform for a specific dataset or group, independent of the data date range.
| Field | Description |
| ---------------- | ---------------------------------------- |
| **Trial Period** | Start and end dates for the trial window |
Outside the trial window, queries against the restricted dataset or group are denied even if other entitlements exist.
### Data Query Range
Controls what type of data the user can query and over what time horizon.
| Type | Behavior |
| -------------------------- | -------------------------------------------------------------------------------- |
| **Historical Only** | Live/real-time data is blocked; only historical queries are allowed |
| **Live Only** | Only live/real-time data is accessible |
| **Both Historical & Live** | No restriction on query mode |
| **Fixed Date Range** | Queries are limited to a specific start/end date range |
| **Rolling Window** | Queries are limited to the last N days from the current date (**Days Lookback**) |
### Limits and Settings
Caps query scope and restricts the symbol universe and visible columns.
| Field | Description |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Max Rows** | Maximum number of rows returned per query |
| **Max Query Days** (2.4+) | Maximum number of calendar days a single query's `back_to`–`as_of` window may span. Use this on high-volume datasets so one oversized request cannot stall other users. |
| **Max Queries** | Maximum number of queries allowed (usage cap) |
| **Max Keys** | Maximum number of keys (symbols) that can be queried in a single request |
| **Keys** | Allow-list of specific keys. Enter individually or paste comma-separated values |
| **Columns** | Allow-list of column names from the dataset schema. Search and multi-select from available columns |
System columns (`_seq`, `timestamp`, `muts`, `symbol`) are excluded from the column picker because they are always required for query execution.
## Dataset vs group rules
Fine-grained permissions can target either:
* **A single dataset** — restrictions apply only to that dataset
* **A dataset group** — restrictions apply to the group entitlement as a whole
Group-level rules are configured the same way: select the group row, click the gear icon, and set restrictions. The dialog title shows the group name.
## Removing fine-grained permissions
To remove all restrictions for a dataset or group:
1. Open the fine-grained settings (gear icon).
2. Disable all three sections (**Trial Range**, **Data Query Range**, **Limits and Settings**), or clear all restriction values.
3. Click **Done**, then **Save Changes**.
Alternatively, revoke the entire dataset or group entitlement — removing an entitlement automatically removes its associated fine-grained rules.
## Interaction with connection entitlements
Datasets inherited from a **connection-level** entitlement cannot have individual fine-grained rules configured in the per-user permissions UI. The gear icon is hidden for locked (inherited) datasets.
To apply fine-grained restrictions in this case, either:
* Assign a direct dataset entitlement instead of (or in addition to) the connection entitlement, or
* Manage access at the connection level through connection permission settings
## How restrictions are enforced
When a user submits a query, the entitlements service evaluates:
1. Whether the user has a dataset, group, or connection entitlement
2. Whether any fine-grained rules apply to that dataset
3. Whether the query parameters (date range, day span, keys, columns, row count) fall within the allowed bounds
If a query violates a restriction, it is rejected before data is returned. In Liberator **2.4** and later, the error names the limit that was exceeded (for example max rows, max query days, or missing entitlement) and the configured value. Callers can correct the request instead of treating every rejection as "not entitled."
## Example configurations
### 90-day rolling trial
1. Enable **Trial Range** and set the trial period dates.
2. Enable **Data Query Range**, select **Rolling Window**, set **Days Lookback** to `90`.
3. Save.
### Cap a single query's date span (2.4+)
1. Enable **Limits and Settings**.
2. Set **Max Query Days** to the widest window you will allow (for example `7` or `30`).
3. Optionally set **Max Rows** as a second backstop.
4. Save.
### Symbol-restricted research access
1. Enable **Limits and Settings**.
2. Set **Keys** to the allowed symbol list (e.g., `AAPL`, `MSFT`, `GOOG`).
3. Optionally set **Max Rows** to cap result size.
4. Save.
### Column-redacted dataset
1. Enable **Limits and Settings**.
2. Use **Columns** to select only the fields the user should see.
3. Save.
## Checking effective access
Users can verify entitled datasets from Python:
```python theme={null}
liberator.datasets(entitled=True)
```
Fine-grained restrictions are enforced at query time and are not visible in the dataset list. If a query fails due to a restriction, the error message indicates which limit was exceeded and, in Liberator **2.4** and later, the configured cap.
For admin-side verification, open the user's **View** dialog or the **JSON** tab (Super Admin) to inspect the full permissions payload including `fine_grain_permissions`.
# Managing Users & Entitlements
Source: https://knowledge.cloudquant.com/administration/managing-users-and-entitlements
Add, create, deactivate users and grant or revoke dataset entitlements in the CloudQuant Data Liberator admin portal
# Managing users and entitlements
This guide covers the full user lifecycle in the CloudQuant Data Liberator admin portal — from onboarding a new account to assigning and revoking dataset access.
## Prerequisites
* **Super Admin** role for creating users, editing details, assigning entitlements, and deactivating users
* **Admin** or **Super Admin** role to add existing user accounts to the permissions system
Navigate to **Users** in the admin portal sidebar.
## User lifecycle
```mermaid theme={null}
flowchart LR
A[Create or provision account] --> B[Add to Liberator]
B --> C[Assign entitlements]
C --> D[Optional fine-grained rules]
D --> E[User queries data]
E --> F[Deactivate when offboarded]
F --> G[Re-activate if needed]
```
## Creating a new user
Super Admins can create a brand-new account directly from the Users page.
Click **Create User** in the top-right of the Users page.
Fill in the required fields:
* **Username** — unique login identifier
* **Email**
* **First name** and **Last name**
* **Company** — select an existing company or type a new one
* **Password** — minimum 8 characters; use the generate button for a secure password (not required when using invite-via-email)
Click **Create**. The platform account is created.
In Liberator **2.4** and later, create and enroll happen in **one step**. The new user is added to Liberator permissions as part of Create User, so you do not need a separate **Add Users** pass for accounts you just created. Use **Add Users** only for existing platform accounts that are not yet in Liberator.
## Invite via email
In Liberator **2.3** and later, Super Admins can provision users without sharing temporary passwords out of band.
Open **Create User** and turn on **Invite via email**.
Provide username, email, name, and company as usual. Liberator sends the invite to the email address you enter.
The recipient opens the invite link, verifies their email, and sets their own password. Liberator requires a password change on first login for newly invited accounts.
In Liberator **2.4** and later, invite and email-verification messages include the username the recipient should sign in with.
### Send a password-reset email
From the Users table **Actions** menu, select **Send reset email** to trigger a self-service password reset for an existing account. Use this when a user cannot sign in and you do not want to set a temporary password yourself.
## Adding users to Liberator
Before a user can receive dataset entitlements, their account must exist in the Liberator permissions system. Use **Add Users** to enroll accounts that are not yet active in Liberator.
Click **Add Users** on the Users page.
The **Available Users** tab lists enabled accounts that are not yet in the permissions system. Use the column filters to search by name, email, or (in Liberator **2.4** and later) **company**, select one or more users, then click **Add**.
The **Deactivated Users** tab lists previously deactivated accounts. Select users and click **Re-activate** to restore their Liberator enrollment. You will still need to re-assign entitlements if they were removed during deactivation.
You can add multiple users in a single operation. Selected users appear as pills above the table for easy review before confirming.
## Editing user details
Super Admins can update profile information and platform roles from the Users table.
In the **Actions** column, click the edit icon for the target user.
Modify any of:
* First name, last name, email, company
* **User Role** — `User`, `Admin`, or `Super Admin` (you can only assign roles at or below your own level)
* **Password** — optionally set a temporary password (copied to clipboard on save)
Click **Save Changes**.
## Assigning dataset entitlements
Entitlements control which datasets a user can query. Super Admins manage entitlements through the **Data Permissions** dialog.
In the Users table **Actions** column, click the data-permissions icon for the target user.
The permissions table lists all datasets and dataset groups. Check the rows you want to grant:
* **Dataset** rows grant access to a single dataset
* **Group** rows grant access to every dataset in that dataset group
Selected items appear as pills above the table. Remove a pill to revoke that entitlement.
Click **Save Changes** and confirm. Changes take effect immediately for the user's next query.
### Inherited (locked) datasets
If a user has **connection-level** entitlements, all datasets in that connection are automatically included. These datasets:
* Appear with a lock icon in the permissions table
* Show as **(locked)** on their selection pill
* Cannot be individually deselected until the connection entitlement is removed
Connection entitlements are typically managed through connection-level permission settings rather than the per-user dataset table.
### Viewing a user's access
Click the **View** action on any user to open a read-only summary of their connections, datasets, groups, and role. Super Admins also see a **JSON** tab with the raw permissions payload.
## Removing entitlements
To revoke dataset or group access:
1. Open **Data Permissions** for the user.
2. Deselect the dataset or group row in the table, or click the remove button on its pill.
3. Click **Save Changes**.
Removing a dataset entitlement also removes any fine-grained permission rules attached to that dataset. See [Fine-Grained Permissions](/administration/fine-grained-permissions) for details.
Deselecting a dataset group removes the group entitlement but does not affect individually assigned datasets within that group unless you deselect those rows separately.
## Deactivating users
Deactivation removes a user from the active Liberator permissions system. The user account is disabled and the user's permissions row is deleted on the backend.
### Single user
Click the delete icon in the **Actions** column, or open **Data Permissions** and click the trash icon in the modal header.
Confirm the deactivation. The user moves to the **Deactivated Users** tab in the Add Users dialog.
### Bulk deactivation
Super Admins can deactivate multiple users at once:
1. Select users using the checkboxes in the rightmost column.
2. Click **Deactivate Selected**.
3. Type `DEACTIVATE` to confirm.
You cannot deactivate your own account. Bulk deactivation requires Super Admin role.
## Re-activating users
1. Click **Add Users** on the Users page.
2. Switch to the **Deactivated Users** tab.
3. Select the users to restore and click **Re-activate**.
4. Re-assign any dataset entitlements and fine-grained rules that were removed during deactivation.
## Verifying entitlements (end user)
Users can confirm their access from Python:
```python theme={null}
# Datasets the user is entitled to query
liberator.datasets(entitled=True)
# All datasets (including those without access)
liberator.datasets(entitled=False)
```
See [Checking Dataset Access](/python-guide/dataset-access) for schema and detail commands.
## Common tasks
| Task | Where | Required role |
| --------------------------------- | ---------------------------------------------- | -------------------- |
| Create a new account | Users → **Create User** | Super Admin |
| Combined create and enroll (2.4+) | Users → **Create User** | Super Admin |
| Invite via email (2.3+) | Users → **Create User** → **Invite via email** | Super Admin |
| Send password-reset email (2.3+) | Users → **Send reset email** | Super Admin |
| Enroll an existing account | Users → **Add Users** | Admin or Super Admin |
| Grant dataset access | Users → **Data Permissions** | Super Admin |
| Revoke dataset access | Users → **Data Permissions** → deselect | Super Admin |
| Change platform role | Users → **Edit** | Super Admin |
| Deactivate a user | Users → delete action | Super Admin |
| Re-activate a user | Users → **Add Users** → Deactivated tab | Admin or Super Admin |
# Administration Overview
Source: https://knowledge.cloudquant.com/administration/overview
Manage users, dataset entitlements, and fine-grained access controls in the CloudQuant Data Liberator admin portal
# Administration overview
The CloudQuant Data Liberator admin portal lets organization administrators control who can access the platform and which datasets each user can query. Access is managed through **entitlements** — explicit grants to datasets, dataset groups, and connections — with optional **fine-grained permissions** that restrict query scope.
## Who can manage access
The admin portal uses a role hierarchy. Each role can assign roles at or below its own level.
| Role | Description |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **User** | Query entitled datasets. No admin portal access. |
| **Admin** | Manage connections, datasets, and user groups. Can add existing user accounts to the permissions system. |
| **Super Admin** | Full platform administration — create users, edit roles, assign entitlements, configure fine-grained permissions, and deactivate users. |
Most user and entitlement management actions described in this section require **Super Admin**. Admins can view the Users page and add users to Liberator, but editing user details, assigning dataset entitlements, and deactivating users are Super Admin actions.
## Permission model
Access is layered:
1. **Platform role** (`user`, `admin`, `super_admin`) — controls what a person can do in the admin portal.
2. **Connection entitlements** — grant access to all datasets within a connection. Datasets inherited from a connection appear as locked in the permissions UI.
3. **Dataset entitlements** — grant access to individual datasets at **Editor** or **User** level.
4. **Dataset group entitlements** — grant access to a curated bundle of datasets defined in **Dataset Groups**.
5. **Fine-grained permissions** — optional per-dataset or per-group restrictions on date range, query day span, query type, row limits, keys, and columns.
```mermaid theme={null}
flowchart TD
A[User account] --> B[Platform role]
A --> C[Connection entitlements]
A --> D[Dataset entitlements]
A --> E[Dataset group entitlements]
D --> F[Fine-grained permissions]
E --> F
C --> G[Inherited dataset access]
```
### Editor vs user
For datasets and connections, entitlements are assigned at one of two levels:
| Level | Meaning |
| ---------- | ---------------------------------------------------------------------------------------------------------------------- |
| **Editor** | Full access to query the dataset (or all datasets in a connection). |
| **User** | Query access at the usage level. In the UI this is labeled **User**; internally it maps to the usage entitlement tier. |
When a user has connection-level access, all datasets in that connection are inherited automatically. Those datasets cannot be individually removed until the connection entitlement is revoked.
## Admin portal navigation
Super Admins and Admins see these relevant sections in the sidebar:
| Page | Purpose |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Users** | Add users, assign entitlements, configure fine-grained permissions |
| **User Groups** | Manage groups of users for bulk entitlement assignment |
| **Dataset Groups** | Manage bundles of datasets for bulk entitlement assignment |
| **Connections** | Configure data source connections |
| **Datasets** | View and manage individual datasets. In Liberator **2.4** and later, catalog cards show date range, symbol count, status, and last updated. Administrators can configure how often symbol discovery refreshes. |
## Related guides
Describe a source in plain language and auto-fill the Create Dataset wizard
Natural-language usage, user, and entitlement reports in the Admin UI
Super-Admin per-query usage dataset (`usage_log`) via the standard query API
Add, create, deactivate users and grant or revoke dataset access
Restrict query date ranges, row limits, keys, and columns per user
Public vs private datasets and the Publicly Available toggle
Schedule warm Parquet caches in S3 or GCS before users query
Lookup-table enrichment with translate=true
Authentication, network security, and the end-user view of entitlements
How users verify their entitled datasets from Python
# Example Queries
Source: https://knowledge.cloudquant.com/api-reference/concepts/example-queries
Practical examples of CloudQuant Data Liberator queries including last known value, time series, all symbols, live datasets, and stats.
# Example queries
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `--cert-type P12 --cert liberator.pfx` flags shown in the curl examples below are only needed for releases prior to 2.0.
## Last known value
A query without date parameters retrieves the most recent available data.
```python Python theme={null}
import liberator
res = liberator.query(name='daily_bars')
df = liberator.get_dataframe(res)
df[['symbol', 'timestamp', 'open', 'high', 'low', 'close', 'volume']]
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"}
});
var df = liberator.get_dataframe(res);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
```
```javascript JavaScript theme={null}
let params = {
symbols: "AAPL",
name: "daily_bars"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Liberator liberator = new Liberator();
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
}});
for (Liberator.QueryResult res : liberator.Generate()) {
Object data = res.GetData();
if (data instanceof org.apache.arrow.vector.VectorSchemaRoot) {
printRecordBatch((org.apache.arrow.vector.VectorSchemaRoot) data);
} else if (data instanceof javax.json.JsonValue) {
System.out.println((javax.json.JsonValue) data);
}
}
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":["AAPL","IBM"],"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","debug_stream":false}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"name", "daily_bars"s}
}));
(void)arrow::PrettyPrint(*table, arrow::PrettyPrintOptions(0, 1), &std::cout);
```
Rather than displaying the entire result, you can select specific columns or pass a `fields` parameter to include only the data you need. Some key fields are returned automatically.
## Time series queries
Two date parameters control historical data retrieval:
* **`as_of`** — Retrieve data from any past date showing information as known on that date. Defaults to current time. Format: `YYYY-MM-DD HH:MM:SS` (time optional).
* **`back_to`** — Specifies where the returned dataset begins. Reads all data "back to" the specified date. Format: `YYYY-MM-DD HH:MM:SS` (time optional).
If `as_of` and `back_to` are identical, the result will not be a time series.
```python Python theme={null}
df = liberator.get_dataframe(liberator.query(
name='daily_bars',
as_of='2020-11-15',
symbols=['FB', 'AAPL', 'NFLX', 'GOOG', 'MSFT', 'IBM']))
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
}
```
```javascript JavaScript theme={null}
let params = {
back_to: "2021-01-12 00:00:00",
as_of: "2021-01-12 23:59:59",
symbols: "AAPL",
name: "daily_bars"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("back_to", "2020-01-01");
put("symbols", "AAPL");
}});
for (Liberator.QueryResult res : liberator.Generate()) {
Object data = res.GetData();
if (data instanceof org.apache.arrow.vector.VectorSchemaRoot) {
printRecordBatch((org.apache.arrow.vector.VectorSchemaRoot) data);
} else if (data instanceof javax.json.JsonValue) {
System.out.println((javax.json.JsonValue) data);
}
}
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
symbols="AAPL",
as_of="2019-09-15",
back_to="2019-09-10"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","back_to":"2022-12-01","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","debug_stream":false}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
{"as_of", "2021-04-10"s},
{"back_to", "2021-03-10"s}
}));
```
## Query all symbols
Omit the `symbols` parameter to retrieve all symbols in a dataset.
```python Python theme={null}
df = liberator.get_dataframe(liberator.query(
name='daily_bars',
as_of='2020-11-15'))
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
}});
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"name":"daily_bars","as_of":"2020-11-15","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query
```
## Query a live dataset
Set the `as_of` parameter to `"live"` for real-time data streaming.
```python Python theme={null}
res = liberator.query(name='live_nasdaq_composite', as_of='live')
for batch in res:
df = batch.to_pandas()
print(df)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "live_nasdaq_composite"},
{"as_of", "live"},
{"symbols", "AAPL"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
}
```
```cpp C++ theme={null}
Liberator::Result ptr = liberator.query({
{"symbols", std::vector{"AAPL"}},
{"name", "live_nasdaq_composite"s},
{"as_of", "live"s}
});
auto generator = *std::get_if(&ptr);
for (auto res = generator(); res.index(); res = generator())
{
auto batch = *std::get_if<2>(&res);
(void)arrow::PrettyPrint(*batch, arrow::PrettyPrintOptions(0, 1), &std::cout);
}
```
## Query stats as JSON
Setting the `stats` parameter to `"total"` returns a JSON result with the count per symbol instead of data rows.
```python Python theme={null}
res = liberator.query(
name='daily_bars',
as_of='2021-04-10',
back_to='2021-03-10',
symbols=['ES'],
stats='total'
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"},
{"stats", "total"}
});
Console.WriteLine(JsonDocumentToIndentedString(res));
```
```r R theme={null}
print(liberator::query(
symbols="AAPL",
name="daily_bars",
as_of="2019-09-15",
back_to="2019-09-10",
stats="total"
))
```
## Get DataFrame / table
Convert query results into a tabular format native to your language.
```python Python theme={null}
df = liberator.get_dataframe(
liberator.query(name='daily_bars', as_of='2020-11-15', symbols=['AAPL'])
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
var df = liberator.get_dataframe(res);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
```
```r R theme={null}
res = liberator::get_dataframe(liberator::query(
symbols="AAPL",
name="daily_bars",
as_of="2025-01-01",
back_to="2024-12-01"
))
print(res)
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
}));
(void)arrow::PrettyPrint(*table, arrow::PrettyPrintOptions(0, 1), &std::cout);
```
# Listing Datasets
Source: https://knowledge.cloudquant.com/api-reference/concepts/listing-datasets
How to list, filter, and inspect available datasets using the CloudQuant Data Liberator API across all supported languages.
# Listing datasets
The `datasets` function retrieves a JSON-formatted list of all datasets available in CloudQuant Data Liberator, with optional filtering and metadata.
## Parameters
| Parameter | Description | Type | Default |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------- |
| `entitled` | Only retrieve datasets you have access to | Boolean | `false` |
| `schema` | Include dataset descriptions and metadata | Boolean | `false` |
| `details` | Include column-level information (names, descriptions). In Liberator **2.4+**, also includes catalog-card fields: date range, symbol count, status, and last updated. Requires `schema` to also be set. | Boolean | `false` |
| `user` | Your CloudQuant-assigned user identifier | String | — |
| `token` | Your assigned authentication token | String | — |
## List all datasets
```python Python theme={null}
import liberator
res = liberator.datasets()
print(res)
```
```csharp C# theme={null}
Liberator liberator;
var res = liberator.datasets();
Console.WriteLine(JsonDocumentToIndentedString(res));
```
```javascript JavaScript theme={null}
let dataset_params = {
schema: false,
details: true
};
liberator.datasets(dataset_params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Liberator liberator = new Liberator();
Object datasets_id = liberator.datasets(new HashMap<>());
for (Liberator.QueryResult res : liberator.Generate()) {
Object data = res.GetData();
if (data instanceof javax.json.JsonValue) {
System.out.println((javax.json.JsonValue) data);
}
}
```
```r R theme={null}
print(liberator::datasets())
```
```cpp C++ theme={null}
Liberator liberator;
auto ptr = liberator.datasets({});
auto json = *std::get_if<0>(&ptr);
if (json)
{
rapidjson::StringBuffer buf;
rapidjson::Writer writer(buf);
json->Accept(writer);
std::cout << buf.GetString() << std::endl;
}
```
### Example output
```json theme={null}
{
"Intraday": ["SSR_Forward_DSI", "Halt_Production_DSI", "SSR_Production_DSI", "News", "Twitter", "Stocktwits"],
"Monthly": ["Joblink Ticker Mapping", "VerticalKnowledge Indeed"],
"Market Data": ["daily_bars", "minute_bars", "nbbo", "trades", "daily_bars_adjusted"]
}
```
## List entitled datasets only
Filter results to show only datasets matching your access permissions.
```python Python theme={null}
liberator.datasets(entitled=True)
```
```csharp C# theme={null}
var res = liberator.datasets(new Dictionary() {
{"entitled", true}
});
```
```java Java theme={null}
Object datasets_id = liberator.datasets(new HashMap<>() {{
put("entitled", true);
}});
```
```r R theme={null}
print(liberator::datasets(entitled=TRUE))
```
```cpp C++ theme={null}
liberator.datasets({{"entitled", true}});
```
## Include schema and column details
```python Python theme={null}
# Schema only
liberator.datasets(schema=True)
# Schema with column details (2.4+: also date range, symbol count, status, last updated)
liberator.datasets(schema=True, details=True)
```
```csharp C# theme={null}
var res = liberator.datasets(new Dictionary() {
{"schema", true},
{"details", true}
});
```
```r R theme={null}
print(liberator::datasets(schema=TRUE, details=TRUE))
```
```cpp C++ theme={null}
liberator.datasets({{"schema", true}, {"details", true}});
```
## Related
For concordance-backed row counts, symbol counts, and date spans without downloading source rows, see [Dataset metadata](/api-reference/concepts/dataset-metadata) (`liberator.metadata`, Liberator 2.3+).
In Liberator **2.4** and later, `liberator.datasets(schema=True, details=True)` also returns catalog-card health (date range covered, symbol count, status, last updated) so you can check freshness before you query. The same fields appear on dataset cards in the Liberator catalog UI.
# Queries & Working with Large Datasets
Source: https://knowledge.cloudquant.com/api-reference/concepts/queries-large-datasets
Best practices for querying large datasets with the CloudQuant Data Liberator API, including point-in-time and time series query patterns.
# Queries & working with large datasets
The CloudQuant Data Liberator service frequently returns extensive result sets, sometimes reaching millions of rows. These queries can be time-consuming to execute. Since data frequencies vary across datasets, start with narrow timeframes for specific symbols before expanding scope.
Consider running a point-in-time query for one symbol to get an idea of how large your dataset is prior to running other queries.
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `--cert-type P12 --cert liberator.pfx` flags shown in the curl examples below are only needed for releases prior to 2.0.
## Query types
Most dataset queries fall into two categories:
* **Point-in-time queries**
* **Time series queries**
Familiarity with both query types early on will improve your API usage efficiency.
## Point-in-time queries
Omit the `back_to` parameter to receive single point-in-time data for each specified symbol based on the `as_of` date/time. If you also exclude `as_of`, CloudQuant Data Liberator defaults to the current date/time.
```python Python theme={null}
df = liberator.get_dataframe(
liberator.query(
name='daily_bars',
as_of='2020-11-15',
symbols=['AAPL']
)
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2020-11-15"},
{"symbols", "AAPL"}
});
```
```javascript JavaScript theme={null}
let params = {
symbols: "AAPL",
name: "daily_bars",
as_of: "2020-11-15"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("symbols", "AAPL");
}});
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
as_of="2020-11-15",
symbols="AAPL"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","as_of":"2020-11-15","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL"}},
{"name", "daily_bars"s},
{"as_of", "2020-11-15"s}
}));
```
## Time series queries
Include a `back_to` parameter that precedes your `as_of` value. When `as_of` is excluded, it defaults to the present moment.
```python Python theme={null}
df = liberator.get_dataframe(
liberator.query(
name='daily_bars',
as_of='2020-11-15',
back_to='2018-11-15',
symbols=['FB', 'AAPL', 'NFLX', 'GOOG', 'MSFT', 'IBM']
)
)
```
```csharp C# theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
}
```
```javascript JavaScript theme={null}
let params = {
back_to: "2021-01-12 00:00:00",
as_of: "2021-01-12 23:59:59",
symbols: "AAPL",
name: "daily_bars"
};
liberator.query(params).then(function(results) {
// Your code
});
```
```java Java theme={null}
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("back_to", "2020-01-01");
put("symbols", "AAPL");
}});
```
```r R theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
symbols="AAPL",
as_of="2019-09-15",
back_to="2019-09-10"
)))
```
```bash RESTful (curl) theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","back_to":"2022-12-01","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query
```
```cpp C++ theme={null}
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
{"as_of", "2021-04-10"s},
{"back_to", "2021-03-10"s}
}));
```
When using `back_to` or `as_of` parameters, the time component is always used even if you do not specify it. Therefore, if you say `as_of: "2023-01-15"`, you are actually saying `as_of: "2023-01-15 00:00:00"`. This may affect result precision depending on your data requirements.
# Query Parameters Reference
Source: https://knowledge.cloudquant.com/api-reference/concepts/query-parameters
Complete reference for all parameters accepted by the CloudQuant Data Liberator query function across all SDKs.
# Query parameters reference
This page documents the parameters accepted by the `query` function across all CloudQuant Data Liberator SDKs. Parameter names are consistent across languages; only the types and calling conventions differ.
At least one of the following is required for every query:
* A dataset via `name` **or** a raw `sql` statement.
* A time anchor: `back_to`, `as_of`, `max_lookback` (LKV mode), or `record_limit` (N-Query mode).
Unknown parameters are rejected by the server when strict validation is enabled (the default).
## Time range parameters
| Parameter | Default | Description |
| -------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `as_of` | `None` | End time for the query. `None` means "now". Integer values are interpreted as microseconds since epoch; strings are interpreted as datetimes. In N-Query backward mode the boundary is **exclusive** (`muts < as_of`). |
| `back_to` | `None` | Start time for the query. `None` means as-of only (single point in time, or LKV). In N-Query forward mode the boundary is **inclusive** (`muts >= back_to`). |
| `max_lookback` | (from dataset config) | Used for LKV (last-known-value) queries. Integer values **less than 1,000,000** are treated as a number of partitions to look back; values **greater than or equal to 1,000,000** are treated as a duration in microseconds. May be supplied per-query or inherited from dataset config. |
For LKV mode, omit `back_to` and use `as_of` (or the default "now") together with `max_lookback`. For N-Query mode, provide either `as_of` (backward) or `back_to` (forward) along with `record_limit`.
### Date format
Datetime strings use the format `YYYY-MM-DD HH:MM:SS` (the time portion is optional). When the time portion is omitted, `00:00:00` is assumed — for example, `as_of: "2023-01-15"` is interpreted as `as_of: "2023-01-15 00:00:00"`. If `as_of` and `back_to` are identical, the result is a single point rather than a time series.
## Dataset & symbol parameters
| Parameter | Default | Description |
| ----------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | `None` | Dataset name(s). A **string** selects a single dataset; a **list of strings** triggers a superquery (multi-dataset merge). Required unless `sql` is provided. |
| `symbols` | `None` | Symbol(s) to filter on. Accepts a string, a list of strings, or `None` for all symbols in the dataset. In Liberator **2.4+**, an unescaped `%` in a value is a `LIKE` wildcard (for example `'PWR%'`). Mix exact keys and patterns in one list. Escape a literal percent as `\%`. |
| `keys` | `None` | Synonym for `symbols`. |
| `fields` | `None` | Field names to return. List of strings or `None` for all fields. Mandatory fields cannot be removed. |
| `distinct` | `["symbol"]` | Partition key for LKV queries: the columns used in `PARTITION BY` (for example, `["symbol"]` or `["symbol", "dataset"]`). |
| `translate` | `None` | When set to `true` or `false`, enables or disables [dataset field mapping](/administration/dataset-field-mapping) lookups configured on the dataset. |
| `where` | `None` | Optional SQL predicate appended to generated filters with `AND`. Omit the `WHERE` keyword — supply only the condition (e.g. `"volume > 1000000"`). Not compatible with `sql`. |
| `sql` | `None` | Raw SQL `SELECT` statement. When present, replaces parameter-based query construction (`name`, time range, `where`, etc.). See [Raw SQL](#raw-sql) below. |
## Custom WHERE filters
The `where` parameter adds extra predicates on top of the time-range and symbol filters CloudQuant Data Liberator generates automatically. The value is appended as `AND ()` to the internal query.
| Aspect | Detail |
| --------------------- | ---------------------------------------------------------------------------------------------- |
| **Syntax** | SQL expression only — no leading `WHERE` |
| **Compatible with** | `name`, `back_to`, `as_of`, `symbols`, `fields`, `record_limit`, LKV (`max_lookback`), N-Query |
| **Incompatible with** | `sql` |
| **Column quoting** | Column names matching the dataset schema are auto-quoted when required |
```python Python theme={null}
result = liberator.query(
name="daily_bars",
symbols=["AAPL"],
back_to="2024-01-01",
as_of="2024-07-01",
where="volume > 1000000"
)
```
```csharp C# theme={null}
var result = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"symbols", new[] {"AAPL"}},
{"back_to", "2024-01-01"},
{"as_of", "2024-07-01"},
{"where", "volume > 1000000"}
});
```
```r R theme={null}
result <- liberator::query(
name = "daily_bars",
symbols = c("AAPL"),
back_to = "2024-01-01",
as_of = "2024-07-01",
where = "volume > 1000000"
)
```
For multi-key datasets (e.g. concordance tables with a `dataset` column), `where` can filter on columns beyond `symbol`:
```python theme={null}
result = liberator.query(
name="concordance",
back_to="2024-01-01",
as_of="2025-01-01",
where="dataset = 'my_derived_dataset'"
)
```
## Raw SQL
The `sql` parameter accepts a complete SQL `SELECT` and bypasses automatic query construction. Use it when you need direct control over joins, subqueries, or complex predicates.
### Requirements
| Rule | Detail |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Statement type | Must be a `SELECT` (each arm of `UNION` / `INTERSECT` / `EXCEPT` is validated separately) |
| `WHERE` clause | Required on every `SELECT` arm |
| Time-range predicate | By default, `WHERE` must include `=`, `>`, `>=`, `<`, `<=`, or `BETWEEN` on `muts` or columns listed in `LIBERATOR_SQL_TIME_RANGE_COLUMNS` |
| Dataset references | Quote table names with double quotes: `"my_dataset"` |
| String literals | Use `$$value$$` dollar-quoting for symbol and string filters |
| Timestamps | Express `muts` bounds as microsecond integers |
### Parameters allowed with `sql`
When `sql` is supplied, only auth, transfer, and cache options may accompany it. Non-empty construction parameters (`name`, `back_to`, `as_of`, `where`, `fields`, etc.) raise an error.
Allowed alongside `sql`: `user`, `system`, `token`, `compress`, `json_xfer`, `batch_size`, `debug_stream`, `force_regen`, `dependent_partition_name`, `skip_validation`, `skip_discovery`, `symbols`, `symbol_key_sequences`, `data_key_column`.
Do not combine `sql` with `name`, `where`, `back_to`, `as_of`, or other query-construction parameters. Choose parameter-based queries or raw SQL, not both.
## Query limits & ordering
| Parameter | Default | Description |
| -------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `record_limit` | `None` | When set to a non-zero integer, triggers **N-Query mode**. **Negative** values (e.g. `-200`) apply a per-symbol limit. **Positive** values (e.g. `200`) apply a global clamp (most recent N records across all symbols). Ignored when both `as_of` and `back_to` are provided. |
| `order` | `None` | Result ordering. One of `"asc_strict"`, `"asc_loose"`, `"desc_strict"`, `"desc_loose"`. Default is ascending. |
### N-query mode
N-Query mode is activated by supplying a non-zero `record_limit`.
* **Anchor:** Backward N-Query uses `as_of` (exclusive); forward N-Query uses `back_to` (inclusive). Exactly one anchor is required.
* **Per-symbol vs global:** A negative `record_limit` returns up to N records per symbol; a positive value returns N records total, distributed by recency.
* **Compatibility:** N-Query works with both single-dataset and superquery queries. When both `as_of` and `back_to` are set, `record_limit` is ignored and a standard time-range query is executed.
## Superquery parameters
A **superquery** is triggered when `name` is a **list** of dataset names. It merges and resamples data from multiple datasets into a single result.
| Parameter | Default | Description |
| ---------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `superq_resample_rule` | `"1D"` | Resampling interval. Accepts pandas-style strings (e.g. `"5T"`, `"1D"`) or Liberator-native values: fixed (`1s`–`60s`, `1m`–`240m`, `1d`, `7d`) or calendar (`day`, `wk`, `mo`, `qtr`, `yr`). |
| `superq_fill_forward` | `True` | Fill forward missing values. Only `false` is currently supported when using aggregates. |
| `superq_aggregates` | `False` | Aggregation spec. `False` / `None` disables aggregation; a dict provides per-dataset record-level (`last_timestamp`, `first_timestamp`) and column-level aggregates (see below). |
### Column-level aggregate aliases
| Alias | Meaning | Description |
| ----- | ------- | --------------------------- |
| `o` | first | First value in the bucket |
| `h` | high | Maximum value in the bucket |
| `l` | low | Minimum value in the bucket |
| `c` | last | Last value in the bucket |
| `s` | sum | Sum of values in the bucket |
Record-level aggregates (`last_timestamp`, `first_timestamp`) apply to the entire dataset per bucket. Output columns are named `{dataset}_{column}_{alias}` (e.g. `trades_price_o`, `trades_volume_s`) for column-level aggregates, and `{dataset}__first_timestamp` / `{dataset}__last_timestamp` for record-level aggregates.
**Dict format example:**
```json theme={null}
{
"my_dataset": [
"last_timestamp",
"first_timestamp",
{
"price": ["o", "h", "l", "c"],
"volume": ["s"],
"datetime_utc": []
}
]
}
```
## Authentication & authorization
| Parameter | Default | Description |
| --------- | ------- | --------------------------------------------- |
| `user` | `None` | Authorized user name (used for entitlements). |
| `system` | `None` | System identifier. Typically `"API"`. |
| `token` | `None` | Authentication token. |
The `user` and `token` parameters can be provided per-query or configured once at the SDK level:
* **Python:** Set via the `liberator.json` file in your working directory.
* **JavaScript:** Use `liberator.set_default("query", credentials)` to avoid passing them every time.
* **RESTful:** Include in each request's JSON body.
* **Other SDKs:** Loaded from `liberator.json` in the working directory.
## Connection & transfer
| Parameter | Default | Description |
| ------------ | ------- | ---------------------------------------------- |
| `compress` | `False` | Compress the response on the wire. |
| `json_xfer` | `False` | Use JSON transfer format instead of Arrow IPC. |
| `batch_size` | `25000` | Rows per batch for chunked / streaming output. |
## Cache & derived data
| Parameter | Default | Description |
| -------------------------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `force_regen` | `False` | Bypass caches (e.g. snapfresh, superquery cache) and regenerate results. |
| `dependent_partition_name` | `None` | Partition UUID for dependency tracking (used for cache invalidation). |
| `skip_validation` | `None` | Tri-state override for cache\_manager validation. When unset, the server default from environment is used. |
| `skip_discovery` | `None` | Tri-state override for partition discovery. When unset, the server default from environment is used. |
## Streaming & debug
| Parameter | Default | Description |
| ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `force_streaming` | `False` | Enable incremental streaming for lower time-to-first-byte. Supported only for single-table, non–N-Query reads. |
| `debug_stream` | `False` | Enable debug / progress output. Language-specific type (e.g. `stderr()` in R, `std::ostream*` in C++, `System.IO.TextWriter` in C#). |
| `warning_stream` | `None` | Output stream for warning information. Same type conventions as `debug_stream`. |
## SDK connection options
These options configure the SDK client rather than the query itself.
| Parameter | Default | Description |
| --------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `url` | From Profile download | CloudQuant Data Liberator server URL. Preconfigured in clients downloaded from your Liberator Profile page. Override for a specific IP or port, e.g. `http://127.0.0.1:47753`. |
## Example queries
### Basic time-range query
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL", "MSFT"],
back_to="2025-01-01 00:00:00",
as_of="2025-01-02 00:00:00",
fields=["Price", "Volume"],
order="asc_strict"
)
```
### Symbol pattern filter (2.4+)
```python theme={null}
result = liberator.query(
name="energy_contracts",
symbols="PWR%", # LIKE wildcard; mix with exact keys in a list if needed
back_to="2026-08-01",
as_of="2026-08-02",
)
```
### LKV (last-known-value) query
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
max_lookback=3 # last 3 partitions
)
```
### N-query: most recent 200 records (backward from `as_of`)
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
record_limit=200,
as_of="2025-01-15 00:00:00"
)
```
### N-query: first 200 records (forward from `back_to`)
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
record_limit=200,
back_to="2025-01-01 00:00:00"
)
```
### N-query per-symbol (negative limit)
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL", "MSFT"],
record_limit=-10, # up to 10 records per symbol
as_of="2025-01-15 00:00:00"
)
```
### Superquery with aggregates
```python theme={null}
result = liberator.query(
name=["dataset1", "dataset2"],
symbols=["AAPL"],
back_to="2021-01-01",
as_of="2021-03-01",
superq_resample_rule="5T",
superq_fill_forward=False,
superq_aggregates={
"dataset1": [
"last_timestamp",
"first_timestamp",
{
"Trade Price": ["o", "h", "l", "c"],
"volume": ["s"]
}
]
}
)
```
### Raw SQL
```python theme={null}
result = liberator.query(
sql='SELECT * FROM "my_dataset" WHERE muts >= 1704067200000000 AND muts < 1704153600000000 AND symbol = $$AAPL$$',
user="my_user",
system="API"
)
```
### Parameter-based query with `where`
```python theme={null}
result = liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2025-01-01 00:00:00",
as_of="2025-01-02 00:00:00",
where="volume > 0",
fields=["Price", "Volume"],
order="asc_strict"
)
```
# C++ SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/cpp/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator C++ API.
# C++ SDK getting started
The CloudQuant Data Liberator C++ SDK uses Apache Arrow for high-performance data streaming with a generator-based iteration pattern.
## Required security files
Place the following files from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Type reference
The SDK uses `std::variant` types for flexible argument and return handling:
```cpp theme={null}
using Arg = std::variant,
std::ostream*>;
using Func = std::function,
std::shared_ptr>()>;
using Result = std::variant, Func>;
```
## Quick start
### Query with get\_table
The simplest approach converts results directly into an Arrow Table:
```cpp theme={null}
Liberator liberator;
auto table = liberator.get_table(liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s},
}));
(void)arrow::PrettyPrint(*table, arrow::PrettyPrintOptions(0, 1), &std::cout);
```
### Query with generator pattern
For streaming or large results, use the generator pattern to iterate through record batches:
```cpp theme={null}
Liberator liberator;
Liberator::Result ptr;
ptr = liberator.query({
{"symbols", std::vector{"AAPL", "GOOGL"}},
{"name", "daily_bars"s}
});
auto generator = *std::get_if(&ptr);
for (auto res = generator(); res.index(); res = generator())
{
auto batch = *std::get_if<2>(&res);
(void)arrow::PrettyPrint(*batch, arrow::PrettyPrintOptions(0, 1),
&std::cout);
}
```
# C# SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/csharp/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator C# API.
# C# SDK getting started
The CloudQuant Data Liberator C# SDK streams data as Arrow Record Batches, which can be converted to Microsoft DataFrames.
## Required security files
You need the following files from your Downloads ZIP, placed in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
`liberator.json` must be in your working directory.
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Quick start
### Query and iterate record batches
```csharp theme={null}
Liberator liberator;
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
foreach (RecordBatch batch in res())
{
var df = DataFrame.FromArrowRecordBatch(batch);
Console.WriteLine(DateTime.Now.ToString("hh:mm:ss.fff") + " " + DataFrameUtils.PrettyPrint(df));
}
```
### Get DataFrame directly
```csharp theme={null}
var res = liberator.query(new Dictionary() {
{"name", "daily_bars"},
{"as_of", "2021-04-10"},
{"back_to", "2021-03-10"},
{"symbols", "ES"}
});
var df = liberator.get_dataframe(res);
Console.WriteLine(DataFrameUtils.PrettyPrint(df));
```
# Configuring Your Account
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/configuring-account
How to set up your CloudQuant account credentials in the Excel Plug-In.
# Excel plug-in: configuring your account
Before accessing any data, you need account information from CloudQuant: **User ID**, **Token**, and **Base URL**.
## Getting started
Download the Excel Plug-In from your Liberator Profile page.
In Liberator **2.4** and later, the CloudQuant Excel plugin is also available from the Liberator client UI — you do not need a separately shared installer.
After installation, Excel displays a new **CloudQuant** Ribbon menu item.
## Configuration steps
1. Select the **CloudQuant** tab
2. Select the **Account** icon to open settings
3. Enter your account credentials in the dialog
## Required information
* **User ID**, **Token**, and **Base URL** — provided on your Liberator Profile page (no trailing slash on Base URL)
When you select OK, the application verifies your login credentials. Once verified, the remaining ribbon buttons are enabled so you can create, run, or interact with queries.
# Creating a Query Using the Ribbon Bar Wizard
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/ribbon-bar-wizard
Step-by-step guide to creating queries in the Excel Plug-In using the Ribbon Bar Wizard.
# Excel plug-in: creating a query using the Ribbon Bar wizard
The **Create** icon opens a Wizard that guides you through the new query process.
If the Wizard ribbon is grayed out, open the right side task pane and select "Query". The Wizard will now be active.
## Step-by-step process
### 1. Provide a query name
You can assign a name to your query at the beginning of the dialog. The system provides a default query name if you skip this step, though naming your query is considered best practice.
### 2. Select dataset from the tree
Available datasets are organized hierarchically by group. Double-click the first arrow or select the second arrow to explore group contents.
#### Searching for datasets by name
A search text box lets you locate specific datasets by name rather than browsing the tree structure.
### 3. Select symbols
Symbols are central to most queries. You have three options:
* Type in a symbol manually
* Link symbols to a spreadsheet location
* Retrieve all available symbols
If you do not specify a set of (one or more) symbols, you will receive all the symbols in the dataset.
### 4. Time range or point in time selection
This dialog allows specification of temporal parameters for the query.
Some datasets contain large amounts of data. Excel has limitations on how many rows of data can appear in a spreadsheet tab. It is best practice to query for a specific symbol first.
### 5. Select data location
Select where results should appear. The **Active Cell** option places data at your current spreadsheet location.
Adjust the active cell location by selecting a cell in the spreadsheet before selecting Finish.
### 6. View results
Query results appear as a formatted table within the spreadsheet.
# Creating a Query Using the Pop Out Task Pane
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/task-pane
How to use the Excel Plug-In task pane to create, manage, and run queries.
# Excel plug-in: creating a query using the pop out Task Pane
## The pop out Task Pane
A pop-out task pane appears on the right side of your spreadsheet and can be minimized, opened, or closed.
If the pane is not visible, use the "Restore Task Pane" button in the CloudQuant menu.
## Pane features
### Query tab
The Query tab displays queries you have created, allowing you to edit, re-run, or delete them. Queries are stored in the spreadsheet where they are created, enabling sharing with other licensed users.
The bottom section shows query results. If a query fails, it displays the reason.
### Datasets tab
The Datasets tab lets you create queries directly from the task pane as a shortcut to using the Wizard:
1. **Step 1**: Select your dataset
2. **Step 2**: Name your query, enter symbols, and select your time range
3. **Step 3**: Select data placement — "New Sheet", "Active Cell", or "Replace Table" (for editing existing queries)
When the DATASET pane is selected, ribbon bar options are grayed out. Select the QUERY tab to restore them.
### Resizing the Task Pane
Drag the left frame edge until your cursor becomes a double arrow, then click and drag to resize. Columns can also be resized using double-arrows on column dividers.
### Restoring the pane
Use the restore icon in the CloudQuant ribbon menu if you close the pane entirely.
## Result sets
Results are placed into tables named after your query. If data goes to a new sheet, that sheet is also named after the query.
Give queries specific, simple names. Access named ranges through Excel's Name Manager (accessible via the lightbulb icon).
# Understanding Queries
Source: https://knowledge.cloudquant.com/api-reference/excel-plugin/understanding-queries
Overview of point-in-time and time series query types in the Excel Plug-In.
# Excel plug-in: understanding queries
Queries with CloudQuant Data Liberator allow you to retrieve two distinct types of data: **point-in-time data** or **time series data**.
## Point-in-time data
When requesting point-in-time data, you typically query information for the current moment. However, you can specify any particular point in time to examine how data appeared at that previous moment — known as the "as of" date in CloudQuant Data Liberator terminology.
When reading a dataset with point-in-time queries, the system returns data for your specified time. This lets you retrieve historical snapshots showing what information was known at earlier dates.
Think of point-in-time data structurally: one row for each given trading symbol.
## Time series data
Time series data returns multiple rows for each trading symbol from your specified dataset. For instance, requesting Minute Bar stock prices over a ten-minute period generates 10 data rows per symbol.
# Java SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/java/getting-started
Prerequisites and setup guide for using the Java CloudQuant Data Liberator External API.
# Java SDK getting started
The CloudQuant Data Liberator Java SDK uses Apache Arrow for efficient data streaming and provides a generator-based iteration pattern.
## Prerequisites
```bash Debian/Ubuntu theme={null}
sudo apt-get install ca-certificates-java openjdk-17-jdk openjdk-17-jre maven
```
```bash RHEL/CentOS theme={null}
sudo dnf install ca-certificates java-17-openjdk java-17-openjdk-devel maven
```
Run the `./build_maven.sh` script to build the Maven project and install the project dependencies.
## Required security files
Place the following files from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
`liberator.json` must be in your working directory.
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Quick start
```java theme={null}
Liberator liberator = new Liberator();
Object query_id = liberator.query(new HashMap<>() {{
put("name", "daily_bars");
put("as_of", "2020-11-15");
put("symbols", "AAPL");
}});
for (Liberator.QueryResult res : liberator.Generate()) {
Object res_id = res.GetId();
Object data = res.GetData();
if (data instanceof org.apache.arrow.vector.VectorSchemaRoot) {
org.apache.arrow.vector.VectorSchemaRoot recordBatch =
(org.apache.arrow.vector.VectorSchemaRoot) data;
printRecordBatch(recordBatch);
} else if (data instanceof javax.json.JsonValue) {
javax.json.JsonValue jsonValue = (javax.json.JsonValue) data;
System.out.println(jsonValue);
}
}
```
# JavaScript SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/javascript/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator NodeJS module.
# JavaScript SDK getting started
The CloudQuant Data Liberator NodeJS module provides promise-based access to CloudQuant datasets.
## Installation
Install the `liberator_module` package in your NodeJS project.
## Required security files
Place the following file from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
## Quick start
```javascript theme={null}
const liberator = require('liberator_module').liberator;
const fs = require('fs');
let credentials = {
user: JSON.parse(fs.readFileSync('liberator.json')).user,
token: JSON.parse(fs.readFileSync('liberator.json')).token,
};
liberator.set_default("query", credentials);
let params = {
as_of: "2021-01-12 23:59:59",
back_to: "2021-01-12 00:00:00",
symbols: "AAPL",
name: "daily_bars",
};
liberator.query(params).then(function(results) {
console.log(results);
});
```
Different datasets have varying update frequencies. Some datasets (like earnings reports) update quarterly, while others update intraday. Time series queries may return limited results depending on data availability.
# Advanced - Query Data Using requests.post RESTful API
Source: https://knowledge.cloudquant.com/api-reference/python/advanced-rest-api
How to query CloudQuant Data Liberator data directly using Python's requests.post method with the RESTful API.
# Advanced - query data using requests.post RESTful API
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `cert=cert` argument and `pfx_to_pem` helper shown below are only needed for releases prior to 2.0.
## requests.post from Python to access CloudQuant Data Liberator data
`requests.post` is an HTTP method you can use to send query arguments to the CloudQuant Data Liberator server for processing.
## Post arguments
* **`{base_url}/liberator/query`** — The query endpoint. Use the base URL from your Liberator Profile page download.
* **`data=json.dumps(...)`** - This `json.dumps` section provides the body of the query with these parameters:
* `compress`
* `json_xfer`
* `user`
* `token`
* `name`
* `as_of`
* `back_to`
* `symbols`
* `system`
* **`headers={'Content-Type':'application/json'}`**
* **`cert=cert`** - cert comes from `with pfx_to_pem('liberator.pfx','') as cert:`
## Query the last known value(s)
To get the last known value in any dataset, do not provide the `as_of` or `back_to` arguments. This will give you the last known value for the given dataset and symbols.
## Query for a time series result
Adding the `back_to` argument for any query will give you the time series data all the way back to the specified date.
Every dataset has different data frequencies. Use short time frames until you become familiar with the datasets. Some datasets are quite large, and an over-extended request can return an extremely large amount of data.
## json.dumps components
| Argument | Description | Type | Example |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ------------------------------------ |
| `symbols` | The security trading symbol(s) you wish to query | String, or List | `"symbols":["AAPL","TLT","GOOG"]` |
| `name` | The name of the dataset **(Required)** | String | `"name": "daily_bars"` |
| `as_of` | This value can be any past date so that you can see the data as it was known on the "as of" date. Defaults to now. | String `YYYY-MM-DD HH:MM:SS` (HH:MM:SS optional) | `"as_of":"2020-11-22 19:51:31"` |
| `back_to` | The date where the return dataset should begin. | String `YYYY-MM-DD HH:MM:SS` (HH:MM:SS optional) | `"back_to":"2020-01-01"` |
| `url` | Optional. The URL of the CloudQuant Data Liberator server. Preconfigured in clients downloaded from your Liberator Profile page. | String | `'http://127.0.0.1:47753'` |
| `system` | The name of the authorized system from which you are querying. | String | `"system":"API"` |
| `compress` | The data compression method on the wire. | Boolean | `True` or `False` |
| `json_xfer` | JSON transfer. Usually False. | Boolean (Always False) | `"json_xfer":False` |
| `user` | The user identifier (as assigned by CloudQuant) | String | `"user":"myUserID"` |
| `token` | The user's assigned token | String | `"token":"mypersonal-private-token"` |
## Example: calling the CloudQuant Data Liberator REST API with Python
```python theme={null}
import sys
import zlib
import json
import base64
import requests
import tempfile
import contextlib
import pyarrow as pa
import OpenSSL.crypto
@contextlib.contextmanager
def pfx_to_pem(pfx_path, pfx_password):
'''Decrypts the .pfx file to be used with requests.'''
with tempfile.NamedTemporaryFile(suffix='.pem') as t_pem:
t_pem.close()
f_pem = open(t_pem.name, 'wb')
pfx = open(pfx_path, 'rb').read()
p12 = OpenSSL.crypto.load_pkcs12(pfx, pfx_password)
f_pem.write(OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM, p12.get_privatekey()))
f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, p12.get_certificate()))
ca = p12.get_ca_certificates()
if ca is not None:
for cert in ca:
f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert))
f_pem.close()
yield t_pem.name
base_url = '' # from Liberator Profile page download
compressed_transfer = True
# POST the query and prepare for a stream of single-line JSON replies
with pfx_to_pem('liberator.pfx', '') as cert:
r = requests.post(f'{base_url}/liberator/query',
data=json.dumps({"compress": compressed_transfer, "json_xfer": False,
"user": user, "token": token,
"name": name, "as_of": as_of, "back_to": back_to, "symbols": symbols,
"system": "API"}),
headers={'Content-Type': 'application/json'},
cert=cert,
stream=True)
# Ensure that the request was successful
if r.status_code != 200:
print(r.json())
else:
batches = []
for line in r.iter_lines(4096):
# Show progress
print('.', end='')
sys.stdout.flush()
# The response stream contains informational messages other than "batch"
if line[0:14] == b'{"exception":"':
print('Error: ' + json.loads(''.join([chr(c) for c in line]), encoding='latin-1')['value'])
break
if line[0:10] != b'{"batch":"':
continue
# Cut out the payload from the JSON line without json module overhead
decoded = base64.b64decode(line[10:-3])
# Interpret the payload as an Apache Arrow IPC stream
reader = pa.ipc.open_stream(zlib.decompress(decoded) if compressed_transfer else decoded)
# Accumulate RecordBatch objects in a list
batches.extend([batch for batch in reader])
print('')
if batches:
# Create an Arrow Table view on the batches and convert to a pandas DataFrame
df = pa.Table.from_batches(batches).to_pandas()
if 'muts' in df and '_seq' in df:
# By default, the data is sorted by symbol... re-sort by time
df = df.sort_values(['muts', '_seq'])
df.reset_index(drop=True, inplace=True)
print(df)
```
# CloudQuant Charting
Source: https://knowledge.cloudquant.com/api-reference/python/charting
Python library for creating branded financial visualizations including candlestick charts, histograms, line charts, bar charts, and more.
# CloudQuant charting
CloudQuant Charting is a Python library for creating branded financial visualizations using the CloudQuant Data Liberator API.
**Import statement:**
```python theme={null}
from liberator import charting
```
## Chart types
### Histogram
Generates distribution/histogram charts with CloudQuant branding.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame containing data
* `col` - string name of column to visualize
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - support HTML tags like ``, ``, ` `
* `histnorm` - normalization type: `'percent'`, `'probability'`, `'density'`, or `'probability density'`
* `width`, `height` - dimensions in pixels (min 400px)
* `overlay_df`, `overlay_col` - add comparison series
**Example:**
```python theme={null}
import liberator
from liberator import charting
res = liberator.query(name="daily_bars")
df_bars = liberator.get_dataframe(res)
df_bars = df_bars.sort_values(by=['volume'], ascending=[False])
change = df_bars.close - df_bars.open
df_bars["change"] = change
fig = charting.Histogram(
df=df_bars[0:100],
col="change",
title="Change Px Distribution of last trading day's top 100 High Volume Stocks",
width=800,
height=400,
histnorm="probability"
)
fig.show()
```
### Candlestick
Generates OHLC candlestick charts with optional technical studies and overlays.
**Returns:** Single figure or array of figures (depending on studies)
**Required arguments:**
* `df` - DataFrame with OHLC data (minimum 20 bars)
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - chart labels with HTML support
* `width`, `height` - dimensions (min 700px width, 400px height)
* `open`, `high`, `low`, `close`, `timestamp` - column names
* `entry_px`, `entry_time`, `close_px`, `close_time`, `entry_side` - trade annotations
* `overlay_df`, `overlay_col`, `overlay_col2` - secondary axis data
**Supported Studies (Single Figure):**
`BBANDS`, `DEMA`, `EMA`, `HT_TRENDLINE`, `KAMA`, `MA`, `MAMA`, `MIDPOINT`, `MIDPRICE`, `SAR`, `SAREXT`, `SMA`, `T3`, `TEMA`, `TRIMA`, `WMA`
**Supported Studies (Array of Figures):**
`ADX`, `ADXR`, `APO`, `AROON`, `AROONOSC`, `BOP`, `CCI`, `CMO`, `DX`, `MACD`, `MACDEXT`, `MFI`, `MINUS_DI`, `MINUS_DM`, `MOM`, `PLUS_DI`, `PLUS_DM`, `PPO`, `ROC`, `ROCP`, `ROCR`, `ROCR100`, `RSI`, `STOCH`, `STOCHF`, `STOCHRSI`, `TRIX`, `ULTOSC`, `WILLR`, `ATR`, `MACDFIX`, `NATR`, `TRANGE`
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
from datetime import datetime, timedelta
back_to = (datetime.now() + timedelta(days=-360)).strftime('%Y-%m-%d 23:59:59')
symbol = "SPY"
res = liberator.query(name="daily_bars", symbols=symbol, back_to=back_to)
df_bars = liberator.get_dataframe(res)
res = liberator.query(name="eia_gas_prices", back_to=back_to)
gaspx = liberator.get_dataframe(res)
fig = charting.Candlestick(
df=df_bars,
overlay_df=gaspx,
overlay_col="retail_gas_price",
height=700,
width=800,
xlabel="Date",
title="SPY vs Retail Gas Prices"
)
fig.show()
```
### LineChart
Creates line charts with support for studies and secondary Y-axis.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `cols` - list of column names to plot
* `x_column` - column name for X-axis (defaults to index)
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - chart labels with HTML support
* `width`, `height` - dimensions
* `y2axis_name` - activates secondary Y-axis
* `overlay_df`, `overlay_cols` - secondary axis data
* `study` - technical study name
* `study_columns` - columns for study calculation
* `timeperiod` - study periods
**Supported Studies:**
`BBANDS`, `MAMA`, `DEMA`, `EMA`, `HT_TRENDLINE`, `KAMA`, `MA`, `MIDPOINT`, `MIDPRICE`, `SAR`, `SAREXT`, `SMA`, `T3`, `TEMA`, `TRIMA`, `WMA`
**Example:**
```python theme={null}
import cloudquantcharting
import liberator
res = liberator.query(
symbols='AMZN',
name='Twitter',
back_to='12/22/2020 08:00:00',
as_of='12/22/2020 16:00:00'
)
twitter_df = liberator.get_dataframe(res)
fig = cloudquantcharting.LineChart(
df=twitter_df,
cols=['s-score', 's-buzz', 's', 's-mean', 'sv-score', 's-dispersion', 's-buzz', 's-delta'],
x_column="timestamp"
)
fig.show()
```
### BarChart
Generates bar charts, including stacked bars for repeated X values.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `xcol` - column name for X-axis
* `ycol` - numeric column for Y-axis
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - labels with HTML support
* `width`, `height` - dimensions
* `singlecolor` - boolean for uniform color (defaults `True`)
* `orientation` - `'v'` (vertical) or `'h'` (horizontal)
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
res = liberator.query(name="daily_bars")
df_bars = liberator.get_dataframe(res)
df_bars = df_bars.sort_values(by=['volume'], ascending=[False])
fig = charting.BarChart(
df=df_bars[0:20],
xcol="symbol",
ycol="spread",
title="Spread Price of High Volume Stocks",
width=800,
height=400
)
fig.show()
```
### PieChart
Generates pie/donut charts with customizable hole size.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `labelcol` - column name for labels
* `valuecol` - numeric column for slice size
**Optional arguments:**
* `title` - chart title with HTML support
* `width`, `height` - dimensions
* `hole` - donut hole size (0-1, where 0 = no hole)
* `colors` - color sequence: default CQ palette, or `'reds'`, `'greens'`, `'blues'`
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
values = [
['Jan', '150000', 'USD'],
['Feb', '200000', 'USD'],
['Mar', '330000', 'USD'],
['Apr', '440000', 'USD'],
['May', '555000', 'USD'],
['June', '355000', 'USD'],
['July', '150000', 'USD'],
['Aug', '130000', 'USD'],
['Sept', '330000', 'USD'],
['Oct', '440000', 'USD'],
['Nov', '130000', 'USD'],
['Dec', '330000', 'USD']
]
df = pd.DataFrame(values, columns=["month", "sales", "currency"])
title = "Sales by month"
fig = charting.PieChart(
df=df,
labelcol='month',
valuecol="sales",
title=title,
height=700
)
fig.show()
```
### GroupedBarChart
Generates grouped bar charts comparing multiple numeric columns.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `groups` - column name for grouping
* `values` - list of numeric column names
**Optional arguments:**
* `title`, `group_label`, `value_label` - labels with HTML support
* `width`, `height` - dimensions
* `colors` - color list (cycles through bars)
* `orientation` - `'v'` or `'h'`
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
dataset_name = 'reddit_wallstreetbets_comments'
df = liberator.get_dataframe(liberator.query(name=dataset_name))
df = df.loc[df['vader_body_sentiment_compound'] > 0]
df = df.sort_values(by=['vader_body_sentiment_compound'], ascending=[False])
charting.GroupedBarChart(
df=df[0:5],
groups="symbol",
values=["vader_body_sentiment_pos", "vader_body_sentiment_compound", "textblob_body_sentiment_subjectivity"],
title="WS Bets Sentiment",
group_label="Symbol",
value_label="sentiment score"
)
```
### ScatterPlot
Generates scatter plots with optional bubble sizing and color coding.
**Returns:** `plotly.graph_objs._figure.Figure`
**Required arguments:**
* `df` - pandas DataFrame
* `x_column` - numeric column for horizontal axis
* `y_column` - numeric column(s) for vertical axis
**Optional arguments:**
* `title`, `xlabel`, `ylabel` - labels with HTML support
* `width`, `height` - dimensions
* `size_column` - numeric column for bubble size (negative values show red/green)
* `size_multiplier` - multiplier to adjust bubble sizes
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
dataset_name = 'reddit_wallstreetbets_comments'
df = liberator.get_dataframe(liberator.query(name=dataset_name))
df = df.loc[df['vader_body_sentiment_compound'] > 0]
df = df.sort_values(by=['vader_body_sentiment_compound'], ascending=[False])
fig = charting.ScatterPlot(
df=df[0:200],
x_column="textblob_body_sentiment_subjectivity",
y_column=['vader_body_sentiment_neu'],
title="WS Bets Sentiment",
size_multiplier=.15,
width=800,
height=800
)
fig.show()
```
### addNotes
Adds annotations/callouts to existing charts for highlighting specific data points.
**Returns:** `plotly.graph_objs._figure.Figure`
**Arguments:**
* `thefig` - figure from a CloudQuant charting function
* `notes` - list of dictionaries with `x`, `y`, and `note` keys
**Example:**
```python theme={null}
import liberator
import pandas as pd
from liberator import charting
from datetime import datetime, timedelta
back_to = (datetime.now() + timedelta(days=-2)).strftime('%Y-%m-%d 23:59:59')
symbol = "AAPL"
res = liberator.query(name="minute_bars", symbols=symbol, back_to=back_to)
df_bars = liberator.get_dataframe(res)
fig = charting.Candlestick(df=df_bars, title=symbol)
mydata = []
data = {'x': df_bars.timestamp[7], 'y': df_bars.open[7], 'note': "T1"}
mydata.append(data)
data = {'x': df_bars.timestamp[155], 'y': df_bars.open[155], 'note': "T2"}
mydata.append(data)
data = {'x': df_bars.timestamp[1000], 'y': df_bars.open[1000], 'note': "T3"}
mydata.append(data)
fig = charting.addNotes(fig, mydata)
fig.show()
```
# Python SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/python/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator Python API from your own environment.
# Python SDK getting started
The CloudQuant Data Liberator Python SDK provides the simplest way to query and work with CloudQuant datasets from any Python environment.
## Prerequisites
* Python 3
## Installation
Install the third-party dependencies Liberator needs. The `liberator.py` client itself is not on PyPI — download it from your Liberator Profile page (see [Required security files](#required-security-files) below).
```bash Conda theme={null}
conda create -n liberator python=3.7
conda activate liberator
conda install -c anaconda requests pyopenssl cryptography six pytz
conda install pandas
conda install -c conda-forge httpx async_generator pyarrow=8.0.0
```
```bash pip theme={null}
python -m venv liberator
source liberator/bin/activate # Windows: liberator\Scripts\activate
pip install requests pandas httpx async_generator "pyarrow==8.0.0" pyOpenSSL cryptography six pytz
```
## Required security files
You need the following files from your Downloads ZIP file:
* **`liberator.py`** — The CloudQuant Data Liberator API for Python
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
Place `liberator.py` in the same folder as your code or in your Python `site-packages` directory. Place `liberator.json` (and `liberator.pfx` if needed) in your **working directory**, or set explicit paths before querying:
```python theme={null}
import liberator
liberator.auth = '/path/to/liberator.json'
liberator.pfx = '/path/to/liberator.pfx' # releases prior to 2.0 only
```
`liberator.json` is resolved from your working directory unless you set `liberator.auth`.
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
When using Jupyter Notebooks, set the working directory to the location containing `liberator.json` (and `liberator.pfx` if you are on a release prior to 2.0). See [this reference](https://kegui.medium.com/change-jupyter-notebook-startup-folder-windows-608dfcfdc104) for changing the Jupyter Notebook startup folder on Windows.
## Optional: CloudQuant charting
If using CloudQuant Charting (included with the API), install these additional packages:
```bash Conda theme={null}
conda install plotly numpy
conda install -c conda-forge ta-lib
```
```bash pip theme={null}
pip install plotly numpy TA-Lib
```
`TA-Lib` on pip requires the [ta-lib C library](https://github.com/TA-Lib/ta-lib-python) to be installed first. Conda installs both the library and Python bindings together.
## Quick start
```python theme={null}
import liberator
df = liberator.get_dataframe(
liberator.query(
name='daily_bars',
as_of='2020-11-15',
symbols=['AAPL']
)
)
print(df)
```
# R SDK Getting Started
Source: https://knowledge.cloudquant.com/api-reference/r-language/getting-started
Installation guide and prerequisites for using the CloudQuant Data Liberator R API.
# R SDK getting started
The CloudQuant Data Liberator R package provides functions for querying datasets and converting results to R DataFrames.
## Required security files
Place the following files from your Downloads ZIP in your working directory:
* **`liberator.json`** — Contains your Username and Token
* **`liberator.pfx`** — Security certificate (only required for releases prior to 2.0)
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx` in your working directory.
## Quick start
### Query and get DataFrame
```r theme={null}
res = liberator::get_dataframe(liberator::query(
symbols="AAPL",
name="daily_bars",
as_of="2025-01-01",
back_to="2024-12-01"
))
print(res)
```
### Function reference
The `get_dataframe` function accepts a query result and converts it into a DataFrame:
```r theme={null}
get_dataframe <- function(res, muts_order_batches=TRUE)
```
You can send debug output to stderr:
```r theme={null}
print(liberator::get_dataframe(liberator::query(
name="daily_bars",
symbols="AAPL",
as_of="2019-09-15",
back_to="2019-09-10",
debug_stream=stderr()
)))
```
# RESTful API Getting Started
Source: https://knowledge.cloudquant.com/api-reference/restful/getting-started
Complete guide to querying the CloudQuant Data Liberator RESTful API using curl and standard HTTP tools.
# RESTful API getting started
The CloudQuant Data Liberator API provides RESTful endpoints for querying financial market data. The primary endpoint is `/liberator/query`, which accepts JSON payloads and returns data as JSON objects or Apache Arrow batches.
## Requirements
* `curl`
* `jq`
* `base64`
* Python with `pyarrow` (for Arrow format decoding)
* `pigz` (for compressed Arrow format)
## Authentication
All requests require user credentials and your Liberator base URL (both provided on your Profile page):
```bash theme={null}
URL=
USER=TODO-ADD-YOUR-USERID-HERE
TOKEN=TODO-ADD-YOUR-TOKEN-HERE
```
As of release version 2.0, the P12 client certificate (`liberator.pfx`) is no longer required. The `--cert-type P12 --cert liberator.pfx` flags shown in the examples below are only needed for releases prior to 2.0.
## Quick start: JSON queries
### Current value query
```bash theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":["AAPL","IBM"],"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","debug_stream":false}' \
$URL/liberator/query
```
### Time range query
```bash theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","back_to":"2022-12-01","debug_stream":false}' \
$URL/liberator/query
```
### Point-in-time query
```bash theme={null}
curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API","as_of":"2022-12-15","debug_stream":false}' \
$URL/liberator/query
```
## Advanced: Apache Arrow format
### Without compression
```bash theme={null}
BATCHES=$(curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":false,"compress":false,"symbols":["AAPL"],"back_to":"2023-01-01","debug_stream":false,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query | jq -r '.[] | select(type=="object") | .batch')
for BATCH in $BATCHES; do
echo $BATCH | base64 -d | python -c "import sys; import pyarrow as pa; print(pa.ipc.open_stream(sys.stdin.buffer.read()).read_next_batch().to_pylist())"
done
```
### With compression
```bash theme={null}
BATCHES=$(curl -qs --cert-type P12 --cert liberator.pfx \
-H "Content-Type: application/json" --noproxy '*' \
--data '{"json_xfer":false,"compress":true,"symbols":["AAPL"],"back_to":"2023-01-01","debug_stream":false,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
$URL/liberator/query | jq -r '.[] | select(type=="object") | .batch')
for BATCH in $BATCHES; do
echo $BATCH | base64 -d | pigz -dz | python -c "import sys; import pyarrow as pa; print(pa.ipc.open_stream(sys.stdin.buffer.read()).read_next_batch().to_pylist())"
done
```
## Response format
Standard JSON response includes fields such as: `_seq`, `_dsname`, `timestamp`, `symbol`, `open`, `high`, `low`, `close`, `volume`, `vwap`, `spread`, `bidvol`, `askvol`, and `count`.
# Data Catalog
Source: https://knowledge.cloudquant.com/data-catalog/overview
Browse 70+ integrated financial, alternative, and economic datasets available through CloudQuant Data Liberator
# Data catalog - data integrations
CloudQuant Data Liberator integrates with over 70 data providers, offering a comprehensive catalog of financial, alternative, and economic datasets.
## Market & financial data
| Dataset | Description |
| -------------------------------- | ------------------------------- |
| **SpiderRock Options & Futures** | Options and futures market data |
| **CryptoQuote** | Cryptocurrency market data |
| **S\&P Dow Jones Indices** | Index data and analytics |
| **Canari Options** | Options analytics and data |
## Alternative data
| Dataset | Description |
| --------------------------------------- | ------------------------------------ |
| **Alexandria** | Earnings and news sentiment analysis |
| **BMLL Technologies** | Level 3 order book data |
| **Exegy Hidden Order Flow** | Hidden and dark pool order flow |
| **GWI Audience Insight** | Consumer audience analytics |
| **Likefolio Twitter** | Social media sentiment from Twitter |
| **Lunarcrush Crypto Sentiment** | Cryptocurrency social sentiment |
| **Smart Insider (Buybacks)** | Corporate buyback intelligence |
| **New Constructs (Earnings Anomalies)** | Earnings quality analytics |
| **Precision Alpha** | Price prediction signals |
| **Danel SmartScore** | Multi-factor scoring |
## Economic data
| Dataset | Description |
| ------------------------------------------ | ----------------------------- |
| **FRED Release Series** | Federal Reserve Economic Data |
| **Bureau of Economic Analysis GDP** | U.S. GDP data |
| **Bureau of Labor Statistics** | Employment and labor data |
| **U.S. Energy Information Administration** | Energy market data |
| **EuroStat** | European economic statistics |
| **CFTC Commitment of Traders** | Futures positioning data |
## Regulatory & reference data
| Dataset | Description |
| ------------------------------ | --------------------------- |
| **FINRA** | Financial regulatory data |
| **SEC Filing Data** | SEC filings and disclosures |
| **Regalytics Regulatory Data** | Regulatory intelligence |
## Environmental & weather data
| Dataset | Description |
| -------------------- | ----------------------------- |
| **Athenium Weather** | Weather analytics for trading |
| **CustomWeather** | Custom weather data feeds |
| **Ilika Weather** | Weather data and forecasts |
## Additional datasets
| Dataset | Description |
| ---------------------- | ----------------------------------------- |
| **Linkup Jobs** | Job market data |
| **Shipfix Shipping** | Global shipping and freight data |
| **ESG Data Providers** | Environmental, social, governance metrics |
Dataset availability depends on your subscription and entitlements. Use `liberator.datasets()` in Python to see which datasets you have access to. In Liberator **2.4** and later, catalog cards (and `liberator.datasets(schema=True, details=True)`) show date range, symbol count, status, and last updated. Contact CloudQuant to request access to additional datasets.
## Accessing datasets
```python theme={null}
import liberator
# List all datasets you have access to
datasets = liberator.datasets()
print(datasets)
# Query a specific dataset
df = liberator.query(
dataset="your_dataset_name",
symbols=["AAPL"],
as_of="2024-01-01",
back_to="2023-01-01"
)
```
See the [Python Guide](/python-guide/dataset-access) for more details on discovering and querying datasets.
# Aligning Two Datasets Into One
Source: https://knowledge.cloudquant.com/data-science-recipes/aligning-datasets
Strategies for merging two different time series datasets, handling timestamp misalignment, symbol mismatches, and frequency differences.
# Aligning two datasets into one
Merging two different time series datasets into one can be like navigating a minefield — it is one of the trickiest challenges in data science.
## Key challenges
When aligning datasets, you need to consider several critical factors:
1. **Timestamp Availability** — Even if your datasets are timestamped as Daily, you still need to know when the data was available to ensure it can line up.
2. **Symbol Consistency** — Do both datasets contain identical symbols? How do you handle mismatches?
3. **Timeframe Misalignment** — What happens when one dataset operates at 1-minute intervals and another at 5-minute intervals?
4. **Data Expansion Strategy** — When expanding lower-frequency data, should you use first values, last values, or an alternative approach?
5. **Data Aggregation** — When merging in the opposite direction, how do you summarize higher-frequency bars? The choice of average, max, min, or other metrics depends on column content.
## Recommended solution: reindex()
The pandas `reindex()` method allows you to forward fill, back fill, use nearest, or fill with None.
```python theme={null}
df.reindex()
```
For detailed usage, see the [pandas DataFrame.reindex documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reindex.html).
## Alternative: SuperQuery
For datasets within CloudQuant Data Liberator, you can use the **SuperQuery** command to have the system perform the merge automatically. SuperQuery resamples multiple datasets onto a common time axis without manual alignment.
If both of your datasets are available in CloudQuant Data Liberator, SuperQuery is often the simplest approach — it handles the alignment for you. See the [SuperQuery recipe](/data-science-recipes/superquery) for details.
# Extracting Parts of a Date or Time from a Timestamp
Source: https://knowledge.cloudquant.com/data-science-recipes/date-time-extraction
Efficiently extract date and time components from timestamp strings in Python DataFrames using string slicing and pandas methods.
# Extracting parts of a date or time from a timestamp
When working with timestamped market data, you often need to extract specific components like the date, time, or hour. Rather than using slower row-by-row loops, you can use vectorized operations for much better performance.
## String slicing with lambda functions
The primary approach uses `apply()` with lambda functions to extract substrings from timestamp data:
```python theme={null}
df = liberator.get_dataframe(
liberator.query(
name='minute_bars',
symbols='SPY',
as_of='2024-07-26',
back_to='2024-07-25'
)
)
df['Date'] = df.apply(lambda row: str(row.timestamp)[0:10], axis=1)
df['Time'] = df.apply(lambda row: str(row.timestamp)[11:16], axis=1)
df['Hour'] = df.apply(lambda row: str(row.timestamp)[11:13], axis=1)
df['DateTime'] = df.apply(lambda row: str(row.timestamp)[0:16], axis=1)
df['DateHour'] = df.apply(lambda row: str(row.timestamp)[0:13], axis=1)
df[['timestamp', 'Date', 'Time', 'Hour', 'DateTime', 'DateHour']]
```
This creates new columns by slicing the timestamp string at specific positions:
| Column | Slice | Example Output |
| ---------- | --------- | ------------------ |
| `Date` | `[0:10]` | `2024-07-26` |
| `Time` | `[11:16]` | `14:30` |
| `Hour` | `[11:13]` | `14` |
| `DateTime` | `[0:16]` | `2024-07-26 14:30` |
| `DateHour` | `[0:13]` | `2024-07-26 14` |
## Alternative: FLOOR method using Pandas DateTime
You can also use pandas built-in datetime methods:
```python theme={null}
df['seconds'] = df['timestamp'].dt.floor('s')
df['Date'] = pd.to_datetime(df["timestamp"]).dt.date
```
String slicing with lambda functions provides fast processing for extracting timestamp components, making it preferable to manual loops for DataFrame operations.
# Downloading Very Large Datasets
Source: https://knowledge.cloudquant.com/data-science-recipes/large-datasets
Chunk large time series queries into smaller segments to handle network instability and avoid timeouts when downloading market data.
# Downloading very large datasets
For large time series datasets, if you experience network instability, chunk large queries into smaller time segments rather than attempting single monolithic downloads.
## Core strategy
The recommended approach divides large queries into smaller, manageable time-based chunks (typically 30-minute segments). This methodology:
* Reduces network timeout and data corruption likelihood
* Automatically retries failed chunks without losing progress
* Continues downloading even when individual chunks fail
* Provides clear feedback on download progress
## Complete Python implementation
```python theme={null}
import liberator
import pandas as pd
from datetime import datetime, timedelta
from typing import Union, List, Optional
import time
def getdata(dataset: str,
start_date: Union[str, datetime],
end_date: Union[str, datetime],
symbols: Union[str, List[str]],
chunk_minutes: int = 30,
max_retries: int = 3,
retry_delay: float = 1.0,
verbose: bool = True) -> pd.DataFrame:
"""
Download data by breaking the time range into smaller chunks.
Args:
dataset: Dataset name (e.g., 'spiderrock_printsets_indexed_single_tenant')
start_date: Start date/time as string 'YYYY-MM-DD [HH:MM:SS]' or datetime
end_date: End date/time as string 'YYYY-MM-DD [HH:MM:SS]' or datetime
symbols: Single symbol string or list of symbols
chunk_minutes: Minutes per chunk (default 30)
max_retries: Maximum retries per chunk (default 3)
retry_delay: Delay between retries in seconds (default 1.0)
verbose: Print progress messages (default True)
Returns:
pandas.DataFrame: Combined data for the entire time range
"""
# Parse and validate inputs
start_dt = _parse_datetime(start_date)
end_dt = _parse_datetime(end_date)
if isinstance(symbols, str):
symbols = [symbols]
if verbose:
print(f"Downloading {dataset} from {start_dt} to {end_dt} for {len(symbols)} symbols")
print(f"Using {chunk_minutes}-minute chunks")
# Generate time chunks
chunks = _generate_time_chunks(start_dt, end_dt, chunk_minutes)
if verbose:
print(f"Processing {len(chunks)} time chunks...")
# Download each chunk
all_dfs = []
failed_chunks = []
empty_chunks = []
for i, (chunk_start, chunk_end) in enumerate(chunks):
if verbose:
end_display = chunk_end.strftime('%H:%M:%S') if chunk_end.second != 0 else chunk_end.strftime('%H:%M')
print(f" Processing chunk {i+1}/{len(chunks)} ({chunk_start.strftime('%H:%M')} - {end_display})", end="")
# Download chunk with retries
result = _download_chunk_with_retry(
dataset=dataset,
start_time=chunk_start,
end_time=chunk_end,
symbols=symbols,
max_retries=max_retries,
retry_delay=retry_delay,
verbose=verbose
)
if result is None:
# Actual failure (exception occurred)
failed_chunks.append((chunk_start, chunk_end))
if verbose:
print(" - FAILED")
elif len(result) == 0:
# Empty result (no data for this time period)
empty_chunks.append((chunk_start, chunk_end))
if verbose:
print(" - EMPTY")
else:
# Success with data
all_dfs.append(result)
if verbose:
print(f" - SUCCESS ({len(result):,} rows)")
# Report results
if verbose:
total_rows = sum(len(df) for df in all_dfs)
print(f"Complete: {len(all_dfs)} chunks with data, {len(empty_chunks)} empty chunks, {len(failed_chunks)} failed chunks")
print(f"Total rows downloaded: {total_rows:,}")
if len(failed_chunks) > 0:
print(f"Warning: {len(failed_chunks)} chunks failed due to errors")
# Combine all dataframes
if all_dfs:
combined_df = pd.concat(all_dfs, ignore_index=True)
# Sort by time if columns exist
if 'muts' in combined_df.columns and '_seq' in combined_df.columns:
combined_df = combined_df.sort_values(['muts', '_seq'])
combined_df.reset_index(drop=True, inplace=True)
elif 'timestamp' in combined_df.columns:
combined_df = combined_df.sort_values('timestamp')
combined_df.reset_index(drop=True, inplace=True)
return combined_df
else:
print("No data downloaded successfully")
return pd.DataFrame()
def _parse_datetime(dt_input: Union[str, datetime]) -> datetime:
"""Parse string or datetime input into datetime object."""
if isinstance(dt_input, datetime):
return dt_input
dt_str = str(dt_input).strip()
# Try different datetime formats
formats = [
'%Y-%m-%d %H:%M:%S.%f', # Full datetime with microseconds
'%Y-%m-%d %H:%M:%S', # Full datetime
'%Y-%m-%d %H:%M', # Date with hour:minute
'%Y-%m-%d', # Date only
]
for fmt in formats:
try:
return datetime.strptime(dt_str, fmt)
except ValueError:
continue
raise ValueError(f"Unable to parse datetime: {dt_input}")
def _generate_time_chunks(start_dt: datetime, end_dt: datetime, chunk_minutes: int) -> List[tuple]:
"""Generate list of (start, end) datetime tuples for chunks."""
chunks = []
current_start = start_dt
chunk_delta = timedelta(minutes=chunk_minutes)
while current_start < end_dt:
current_end = min(current_start + chunk_delta, end_dt)
chunks.append((current_start, current_end))
current_start = current_end
return chunks
def _download_chunk_with_retry(dataset: str,
start_time: datetime,
end_time: datetime,
symbols: List[str],
max_retries: int,
retry_delay: float,
verbose: bool = True) -> Optional[pd.DataFrame]:
"""Download a single time chunk with retry logic."""
# Format times as strings for liberator
start_str = start_time.strftime('%Y-%m-%d %H:%M:%S')
end_str = end_time.strftime('%Y-%m-%d %H:%M:%S')
for attempt in range(max_retries):
try:
# Query the data
query_result = liberator.query(
name=dataset,
symbols=symbols,
back_to=start_str,
as_of=end_str
)
# Convert to dataframe
df = liberator.get_dataframe(query_result)
# Return the dataframe (could be empty, but that's not a failure)
return df
except Exception as e:
if attempt < max_retries - 1: # Don't sleep on last attempt
time.sleep(retry_delay)
else:
if verbose:
print(f" Failed chunk {start_time.strftime('%H:%M')}-{end_time.strftime('%H:%M')} after {max_retries} attempts: {str(e)}")
return None # Return None only on actual exception
return None
# Example usage functions
def download_single_day(dataset: str, date: str, symbols: Union[str, List[str]], **kwargs) -> pd.DataFrame:
"""
Convenience function to download a single day's data.
Args:
dataset: Dataset name
date: Date as 'YYYY-MM-DD'
symbols: Symbol(s) to download
**kwargs: Additional arguments passed to getdata()
Returns:
pandas.DataFrame: Day's data
"""
start_date = f"{date} 00:00:00"
end_date = f"{date} 23:59:59"
return getdata(
dataset=dataset,
start_date=start_date,
end_date=end_date,
symbols=symbols,
**kwargs
)
def download_date_range(dataset: str,
start_date: str,
end_date: str,
symbols: Union[str, List[str]],
**kwargs) -> pd.DataFrame:
"""
Download data across multiple days.
Args:
dataset: Dataset name
start_date: Start date as 'YYYY-MM-DD'
end_date: End date as 'YYYY-MM-DD'
symbols: Symbol(s) to download
**kwargs: Additional arguments passed to getdata()
Returns:
pandas.DataFrame: Multi-day data
"""
start_dt = _parse_datetime(f"{start_date} 00:00:00")
end_dt = _parse_datetime(f"{end_date} 23:59:59.999")
return getdata(
dataset=dataset,
start_date=start_dt,
end_date=end_dt,
symbols=symbols,
**kwargs
)
```
## Usage examples
### Single day, multiple symbols
```python theme={null}
df1 = download_single_day(
dataset='spiderrock_printsets_indexed_single_tenant',
date='2025-09-08',
symbols=['AAPL', 'GOOGL', 'MSFT']
)
```
### Custom time range
```python theme={null}
df2 = getdata(
dataset='spiderrock_printsets_indexed_single_tenant',
start_date='2025-09-08 09:30:00',
end_date='2025-09-08 16:00:00',
symbols='AAPL',
chunk_minutes=15,
verbose=False
)
```
### Multi-day download
```python theme={null}
df3 = download_date_range(
dataset='spiderrock_printsets_indexed_single_tenant',
start_date='2025-09-03',
end_date='2025-09-06',
symbols=['AAPL', 'GOOGL'],
chunk_minutes=120 # 2-hour chunks for longer periods
)
```
## Sample output
```
Downloading spiderrock_printsets_indexed_single_tenant from 2025-09-03 00:00:00
to 2025-09-06 23:59:59.999000 for 2 symbols
Using 120-minute chunks
Processing 48 time chunks...
Processing chunk 1/48 (00:00 - 02:00) - EMPTY
Processing chunk 2/48 (02:00 - 04:00) - EMPTY
Processing chunk 3/48 (04:00 - 06:00) - EMPTY
Processing chunk 4/48 (06:00 - 08:00) - EMPTY
Processing chunk 5/48 (08:00 - 10:00) - SUCCESS (143,762 rows)
Processing chunk 6/48 (10:00 - 12:00) - SUCCESS (168,790 rows)
...
Complete: 13 chunks with data, 35 empty chunks, 0 failed chunks
Total rows downloaded: 959,576
```
For multi-day downloads, increase `chunk_minutes` to reduce the number of API calls. A value of 120 (2 hours) works well for longer date ranges.
The chunking approach is designed for network resilience. If you are on a stable connection and downloading moderately sized datasets, a single query may be simpler and faster.
# Using Market Calendar (mcal) to Identify Trading Dates
Source: https://knowledge.cloudquant.com/data-science-recipes/market-calendar
Programmatically determine trading dates using the pandas_market_calendars library for NYSE, CME, CBOE, NASDAQ, and other exchanges.
# Using Market Calendar (mcal) to identify trading dates
When retrieving market data, it is beneficial to programmatically determine which dates represent actual trading dates. The **Market Calendar (mcal)** library enables this functionality.
## US stock market calendars
Market Calendar supports multiple US exchanges including NYSE, CME, CBOE, and NASDAQ.
## Implementation example
```python theme={null}
# Get NYSE Schedule for date range
# Documentation: https://pandas-market-calendars.readthedocs.io/en/latest/usage.html
start_date = '2024-01-01'
end_date = '2024-06-01'
import pandas as pd
import pandas_market_calendars as mcal
nyse = mcal.get_calendar('NYSE')
schedule = nyse.schedule(start_date=start_date, end_date=end_date)
schedule
schedule.market_open.iloc[0] # open time first date
schedule.market_close.iloc[0] # close time first date
schedule.market_open.iloc[-1] # open time last date
schedule.market_close.iloc[-1] # close time last date
schedlist = list(schedule.index)
schedlist
```
Although mcal is generally extremely reliable, it is an external library and has on occasion missed a market holiday. Implement defensive coding practices and verify results independently.
The library demonstrates strong reliability for US market holidays but cannot be vouched for with equal confidence regarding international markets.
## Resources
* [Official pandas\_market\_calendars Documentation](https://pandas-market-calendars.readthedocs.io/en/latest/usage.html)
# Melt and Wide to Long - Unpivoting a Pivot
Source: https://knowledge.cloudquant.com/data-science-recipes/melt-and-unpivot
Convert DataFrames from wide to long format using pandas melt and wide_to_long methods.
# Melt and wide to long - unpivoting a pivot
Pandas provides two methods for converting DataFrames from wide to long format (unpivoting).
## Melt method
Unpivot a DataFrame from wide to long format, optionally leaving identifiers set.
```python theme={null}
df.melt()
```
The `melt` function provides flexibility when restructuring data from a wide format into a long format. You can specify which columns serve as identifiers and which become variable-value pairs.
Use `melt` when you need fine-grained control over which columns become identifiers and which become variable-value pairs.
**Documentation:** [pandas.melt](https://pandas.pydata.org/docs/reference/api/pandas.melt.html)
## Wide to long method
Unpivot a DataFrame from wide to long format. Less flexible but more user-friendly than melt.
```python theme={null}
pd.wide_to_long()
```
The `wide_to_long` function offers a more accessible alternative to melt, trading some flexibility for improved usability when performing standard unpivoting operations.
Use `wide_to_long` for simpler, standard unpivoting operations where ease of use is more important than flexibility.
**Documentation:** [pandas.wide\_to\_long](https://pandas.pydata.org/docs/reference/api/pandas.wide_to_long.html)
# A Merge or Join
Source: https://knowledge.cloudquant.com/data-science-recipes/merge-and-join
Combine two DataFrames by columns using pandas merge, join, and merge_asof operations.
# A merge or join
A merge or join operation expands the number of **columns** in a resulting DataFrame by combining data from two tables based on shared values. This differs from concatenation, which combines **rows**.
## Merge vs. concat
| Operation | What it does |
| -------------- | -------------------------------------------------------------------------------------------------------------------- |
| **CONCAT** | Combines rows from two DataFrames (e.g., two 10-row DataFrames yield 20 rows). Generally requires identical columns. |
| **MERGE/JOIN** | Combines columns by matching rows on a shared column, typically increasing column count. |
## pd.merge()
Merging requires a **left** dataset, a **right** dataset, and a common column specified with the `on` parameter.
### Join types
The `how` parameter controls which rows appear in results:
| Join Type | Behavior |
| --------- | ------------------------------------------------------------------------ |
| `inner` | **(DEFAULT)** Only rows where the merge column exists in both DataFrames |
| `left` | All rows from the left DataFrame; `NaN` where right data is missing |
| `right` | All rows from the right DataFrame; `NaN` where left data is missing |
| `outer` | All rows from both DataFrames; `NaN` where matches do not exist |
Use `indicator=True` to add a column showing the merge source (`'both'`, `'left_only'`, `'right_only'`) — especially useful when debugging outer joins.
### Basic examples
```python theme={null}
# Standard merge on a shared column
res = pd.merge(left, right, on='shared_column')
# Left join
res = pd.merge(left, right, on='shared_column', how='left')
# Merge with different column names
res = pd.merge(left, right, left_on='leftcol2',
right_on='rightcol7', how='left')
# Select specific columns from right before merging
res = pd.merge(left, right[['col1', 'col2', 'col3']],
on='shared_column')
# Check overlap before merging
left['datesym'].isin(right['datesym']).value_counts()
```
## merge\_asof()
`merge_asof` offers additional flexibility for time-based merging:
* Merging on multiple values
* Specifying tolerance thresholds
### Example: matching quotes to trades
```python theme={null}
pd.merge_asof(trades, quotes, on='time', by='ticker')
# With tolerance
pd.merge_asof(trades, quotes, on='time', by='ticker',
tolerance=pd.Timedelta('2ms'))
```
If `merge_asof()` fails, try removing the `tolerance` parameter.
## Reference
* [pandas.merge\_asof documentation](https://pandas.pydata.org/docs/reference/api/pandas.merge_asof.html)
# Summarizing Data with a Pivot Table
Source: https://knowledge.cloudquant.com/data-science-recipes/pivot-tables
Use pandas pivot tables to summarize, reorganize, and explore large market datasets by aggregating data into meaningful categories.
# Summarizing data with a pivot table
A pivot table is a data analysis tool that enables you to summarize, reorganize, and explore large datasets. It aggregates data, rotates perspectives, and filters, sorts, and groups information into meaningful categories.
## Key benefits
* **Simplify Complex Data** — Break down large datasets into manageable summaries
* **Identify Trends** — Reveal patterns not immediately obvious in raw data
* **Flexible Analysis** — Quickly change layout and structure to answer different questions
* **Efficiency** — Automate data analysis for faster, regular reporting
## Pandas pivot table syntax
```python theme={null}
import pandas as pd
pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'], aggfunc='sum')
```
## Practical example: analyzing odd lots in AAPL trades
This example demonstrates analyzing AAPL trade data by:
1. Extracting trades between 9:30 AM and 4 PM
2. Creating an `odd_lots` boolean column (shares \< 100)
3. Extracting hour from timestamp
4. Pivoting to count shares by hour and lot size
5. Calculating percentage of odd lots per hour
```python theme={null}
import liberator
%time df = liberator.get_dataframe(
liberator.query(
name='trades',
symbols='AAPL',
as_of='2024-07-30 15:59:58',
back_to='2024-07-30 09:30:01'
)
)
df['odd_lots'] = df['shares'] < 100
df['Hour'] = df.apply(lambda row: str(row.timestamp)[11:13], axis=1)
pivot_table = df.pivot_table(
values='shares',
index='Hour',
columns='odd_lots',
aggfunc='sum',
fill_value=0
)
pivot_table['% Odd Lots'] = (
pivot_table[True] / (pivot_table[True] + pivot_table[False]) * 100
)
pivot_table
```
## Parameter reference
| Parameter | Purpose |
| ------------ | -------------------------------------------------------- |
| `aggfunc` | Specifies aggregation function (`'sum'`, `'mean'`, etc.) |
| `values` | Target column for aggregation |
| `index` | Column(s) defining row groupings |
| `columns` | Column(s) defining column splits |
| `fill_value` | Default value for missing data |
# Python Data Science Shortcuts and Snippets
Source: https://knowledge.cloudquant.com/data-science-recipes/shortcuts-and-snippets
Practical Python solutions for common data science challenges when working with DataFrames and market data.
# Python data science shortcuts and snippets
The solutions to most challenges in data science are best solved by experimentation and failure. This guide focuses on practical, daily-use solutions for common data science tasks.
Large Language Models like ChatGPT can help format data effectively when time is limited. They can be a valuable companion for quick data transformations.
This collection of shortcuts and snippets covers the most frequently encountered data manipulation tasks. While it does not provide comprehensive training on data science fundamentals or Lambda functions, these tools can rapidly transform large DataFrames.
The recipes in this section cover specific topics in depth. Use the navigation to explore individual techniques like date extraction, pivot tables, merging datasets, and more.
# SuperQuery
Source: https://knowledge.cloudquant.com/data-science-recipes/superquery
Resample pandas DataFrames from CloudQuant Data Liberator into a common time axis by querying multiple datasets simultaneously.
# SuperQuery
SuperQuery enables you to resample pandas DataFrames from CloudQuant Data Liberator into a common time axis by querying multiple datasets simultaneously.
## Initial setup
```python theme={null}
import liberator
symbols = ['TSLA']
as_of = '2024-06-07'
back_to = '2024-06-04'
%time df1 = liberator.get_dataframe(
liberator.query(
symbols=symbols,
name='minute_bars',
as_of=as_of,
back_to=back_to
)
)
liberator.get_dataframe(
liberator.query(
name='daily_bars',
symbols=symbols,
as_of=as_of,
back_to=back_to
)
)
```
This loads two independent datasets: minute bars and daily bars.
## Querying multiple datasets together
SuperQuery lets you specify how multiple datasets are reindexed and merged together using the `superq_resample_rule` parameter.
### Daily (1D) resampling
For a three-day query using daily frequency, the result contains three rows with daily bars and the final minute bar of each day:
```python theme={null}
df = liberator.get_dataframe(
liberator.query(
symbols=symbols,
name=['daily_bars', 'minute_bars'],
as_of=as_of,
back_to=back_to,
superq_periods_per_batch_override=5,
superq_resample_rule='1D'
)
)
df
```
### Hourly (60T) resampling
For a three-day query using 60-minute frequency, the result contains 24 rows per day. Daily bar data is timestamped at 8pm:
```python theme={null}
%time df4 = liberator.get_dataframe(
liberator.query(
symbols=symbols,
name=['daily_bars', 'minute_bars'],
as_of=as_of,
back_to=back_to,
superq_periods_per_batch_override=5,
superq_resample_rule='60T'
)
)
df
```
## The superq\_resample\_rule parameter
The `superq_resample_rule` parameter uses pandas "Offset Aliases" to define the resampling frequency.
Common values:
| Rule | Frequency |
| ----- | ---------- |
| `1T` | 1 minute |
| `5T` | 5 minutes |
| `15T` | 15 minutes |
| `60T` | 60 minutes |
| `1D` | 1 day |
For a complete list of offset aliases, see the [pandas time series offset aliases documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases).
# Azure Blob Storage
Source: https://knowledge.cloudquant.com/datasource-config/azure-blob
Configure Azure Blob Storage datasources
# Azure Blob Storage
Azure Blob Storage datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from Azure Blob containers. CloudQuant Data Liberator mounts the container via FUSE.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest on Azure Blob, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | -------------------------- |
| `connection_type` | string | Must be `"azure"` |
| `account` | string | Azure storage account name |
| `key` | string | Storage account access key |
| `container` | string | Blob container name |
### Optional fields
| Field | Type | Default | Description |
| ------------- | ------ | ------- | ------------------------------------------------------------ |
| `endpoint` | string | | Blob endpoint URL (override for sovereign clouds or Azurite) |
| `prefix` | string | `""` | Blob prefix (virtual directory) within the container |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
| `object_key` | string | | Object key pattern for blob selection |
The `endpoint` field is only needed for non-standard Azure environments such as Azure Government, Azure China, or local development with Azurite. For standard Azure public cloud, CloudQuant Data Liberator derives the endpoint from the account name.
### Example connection
```json theme={null}
{
"name": "azure-market-data",
"connection_type": "azure",
"account": "mystorageaccount",
"key": "base64encodedstorageaccountkey==",
"container": "market-data",
"prefix": "equities/daily/"
}
```
Storage account keys grant full access to the entire storage account. Rotate keys regularly and use environment variables or a secrets manager to inject them at deployment time.
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "azure-eod-prices",
"connection_type": "azure",
"account": "caboretlstorage",
"key": "xYzAbCdEfGhIjKlMnOpQrStUvWxYz0123456789==",
"container": "eod-data",
"prefix": "prices/us-equities/"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-eod",
"connection": "azure-eod-prices",
"data_args": {
"file_pattern": "eod_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "date",
"data_dt_format": "%Y-%m-%d",
"data_dt_timezone": "America/New_York",
"data_key_column": "ticker",
"fname_dt_regex": "eod_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "ticker", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "date", "type": "string", "group": "time", "description": "Trade date" },
{ "name": "open", "type": "double", "group": "value", "description": "Open price" },
{ "name": "high", "type": "double", "group": "value", "description": "High price" },
{ "name": "low", "type": "double", "group": "value", "description": "Low price" },
{ "name": "close", "type": "double", "group": "value", "description": "Close price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Daily volume" },
{ "name": "adj_close", "type": "double", "group": "value", "description": "Adjusted close price" }
]
}
```
## Azure setup requirements
Ensure the following before configuring your Azure Blob connection:
1. **Storage account** exists and is accessible from the CloudQuant Data Liberator host
2. **Container** is created within the storage account
3. **Access key** is available (found under Storage Account > Access Keys in the Azure Portal)
4. **Network access** is permitted — if the storage account uses firewall rules, add the CloudQuant Data Liberator host's IP to the allowed list
For initial testing, use Azure Storage Explorer to verify your files are accessible before configuring the CloudQuant Data Liberator connection. This eliminates credential and network issues early.
See the [Azure Blob Storage integration guide](/integrations/azure-blob-storage) for additional setup details.
# CIFS/SMB
Source: https://knowledge.cloudquant.com/datasource-config/cifs
Configure Windows/Samba network share datasources
# CIFS/SMB
CIFS/SMB datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from Windows file shares or Samba servers. This is commonly used in enterprise environments where data is published to shared network drives.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest from CIFS/SMB shares, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ---------- | ------ | ---------------------------------------------------------------------- |
| `username` | string | Windows/Samba domain username |
| `password` | string | Password for authentication |
| `server` | string | Server hostname or IP address |
| `path` | string | UNC path to the share (e.g., `"sharename"` or `"sharename/subfolder"`) |
The `connection_type` for CIFS/SMB connections is inferred from the presence of the `server` and `path` fields. Unlike other connection types, there is no explicit `connection_type` field required.
### Optional fields
| Field | Type | Default | Description |
| -------- | ------ | ------- | -------------------------------------------- |
| `prefix` | string | `""` | Subdirectory within the share to use as root |
### Example connection
```json theme={null}
{
"name": "cifs-shared-drive",
"username": "DOMAIN\\datauser",
"password": "w1nd0wsP@ss",
"server": "fileserver.corp.example.com",
"path": "MarketData",
"prefix": "daily/equities/"
}
```
Use a dedicated service account with read-only permissions on the share. Avoid using personal credentials, as password changes would break the connection.
### Domain authentication
For Active Directory environments, include the domain in the username:
```json theme={null}
{
"username": "CORP\\svc-liberator"
}
```
Or use the UPN format:
```json theme={null}
{
"username": "svc-liberator@corp.example.com"
}
```
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "cifs-risk-reports",
"username": "CORP\\svc-liberator",
"password": "s3rv1ceAcct!",
"server": "nas01.corp.example.com",
"path": "RiskReports",
"prefix": "daily/"
}
```
### Dataset
```json theme={null}
{
"name": "daily-risk-metrics",
"connection": "cifs-risk-reports",
"data_args": {
"file_pattern": "risk_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "report_date",
"data_dt_format": "%Y-%m-%d",
"data_dt_timezone": "America/New_York",
"data_key_column": "portfolio_id",
"fname_dt_regex": "risk_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "portfolio_id", "type": "string", "group": "key", "description": "Portfolio identifier" },
{ "name": "report_date", "type": "string", "group": "time", "description": "Report date" },
{ "name": "var_95", "type": "double", "group": "value", "description": "95th percentile VaR" },
{ "name": "var_99", "type": "double", "group": "value", "description": "99th percentile VaR" },
{ "name": "cvar", "type": "double", "group": "value", "description": "Conditional VaR" },
{ "name": "beta", "type": "double", "group": "value", "description": "Portfolio beta" },
{ "name": "sharpe", "type": "double", "group": "value", "description": "Sharpe ratio" }
]
}
```
## Network requirements
| Requirement | Detail |
| ------------------ | ------------------------------------------------------------------------ |
| **Outbound port** | TCP port 445 (SMB) to the file server |
| **DNS resolution** | The server hostname must resolve from the CloudQuant Data Liberator host |
| **Authentication** | NTLM or Kerberos (depending on domain configuration) |
| **SMB version** | SMB 2.0 or higher recommended; SMBv1 is deprecated and insecure |
If the CloudQuant Data Liberator host is Linux-based, install `cifs-utils` for CIFS mount support.
```bash Debian/Ubuntu theme={null}
apt install cifs-utils
```
```bash RHEL/CentOS theme={null}
yum install cifs-utils
```
## Troubleshooting
| Issue | Possible Cause | Solution |
| ------------------------------------ | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Mount fails with "Permission denied" | Incorrect credentials or insufficient share permissions | Verify credentials and ensure the service account has read access to the share |
| Mount fails with "Host is down" | Network connectivity or firewall blocking port 445 | Check firewall rules and verify SMB port is reachable |
| Files not found | Incorrect `path` or `prefix` | Verify the UNC path using `smbclient` or Windows Explorer |
| Encoding errors | Files use non-UTF-8 encoding | Set `encoding` in data\_args (common alternatives: `"latin-1"`, `"cp1252"` for Windows-origin files) |
# FTPS
Source: https://knowledge.cloudquant.com/datasource-config/ftps
Configure FTP over TLS/SSL datasources
# FTPS
FTPS (FTP over TLS/SSL) datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from FTP servers secured with TLS encryption. This is commonly used with legacy financial data feeds and vendor systems that require explicit FTPS connections.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest over FTPS, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | --------------------------------- |
| `connection_type` | string | Must be `"ftps"` |
| `host` | string | FTP server hostname or IP address |
| `user` | string | Username for authentication |
### Optional fields
| Field | Type | Default | Description |
| ------------- | ------ | ------- | -------------------------------------------------------------------------------- |
| `port` | int | `21` | FTP control port number |
| `password` | string | | Password for authentication |
| `key` | string | | Client certificate private key (PEM format) for certificate-based authentication |
| `no_verify` | bool | `false` | Skip TLS certificate verification |
| `prefix` | string | `""` | Remote directory path to use as root |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
Setting `no_verify` to `true` disables TLS certificate validation. This should only be used for testing or when connecting to servers with self-signed certificates on trusted internal networks. Never use this in production with external servers.
You must provide either `password` or `key` for authentication. Password authentication is most common for FTPS. Client certificate authentication (`key`) is used when the server requires mutual TLS.
### Example connection
```json theme={null}
{
"name": "ftps-vendor-feed",
"connection_type": "ftps",
"host": "ftp.securefeed.example.com",
"port": 21,
"user": "datafeed",
"password": "v3nd0rP@ss",
"prefix": "/outgoing/daily/"
}
```
### Example connection (self-signed certificate)
```json theme={null}
{
"name": "ftps-internal-data",
"connection_type": "ftps",
"host": "ftp.internal.corp.net",
"port": 990,
"user": "etl-service",
"password": "internalPass123",
"no_verify": true,
"prefix": "/data/exports/"
}
```
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "ftps-reference-data",
"connection_type": "ftps",
"host": "ftps.refdata.example.com",
"port": 21,
"user": "cq-download",
"password": "refDataP@ss",
"prefix": "/reference/securities/"
}
```
### Dataset
```json theme={null}
{
"name": "security-master",
"connection": "ftps-reference-data",
"data_args": {
"file_pattern": "secmaster_*.csv",
"sep_override": "|",
"encoding": "utf-8",
"data_dt_column": "effective_date",
"data_dt_format": "%Y-%m-%d",
"data_dt_timezone": "America/New_York",
"data_key_column": "cusip",
"fname_dt_regex": "secmaster_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "cusip", "type": "string", "group": "key", "description": "CUSIP identifier" },
{ "name": "effective_date", "type": "string", "group": "time", "description": "Effective date" },
{ "name": "ticker", "type": "string", "group": "value", "description": "Ticker symbol" },
{ "name": "company_name", "type": "string", "group": "value", "description": "Company name" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Primary exchange" },
{ "name": "sector", "type": "string", "group": "value", "description": "GICS sector" },
{ "name": "shares_outstanding", "type": "int64", "group": "value", "description": "Shares outstanding" }
]
}
```
## FTPS vs SFTP
| Feature | FTPS | SFTP |
| --------------------- | ------------------------------- | -------------- |
| Protocol | FTP + TLS | SSH subsystem |
| Default port | 21 (explicit) or 990 (implicit) | 22 |
| Firewall friendliness | Requires data port range | Single port |
| Common use | Legacy financial feeds | Modern systems |
| Certificate | TLS/SSL certificates | SSH host keys |
If your data vendor supports both FTPS and SFTP, prefer SFTP. It uses a single port, simplifies firewall configuration, and is generally easier to manage.
## Network requirements
| Requirement | Detail |
| ------------------ | ----------------------------------------------------------------------------------- |
| **Outbound port** | TCP port 21 (control) + passive data port range to the FTP server |
| **Passive mode** | FTP passive mode is required; ensure the passive port range is open |
| **DNS resolution** | The hostname must resolve from the CloudQuant Data Liberator host |
| **TLS support** | Server must support explicit TLS (AUTH TLS) on port 21, or implicit TLS on port 990 |
# Local File (CSV/TSV)
Source: https://knowledge.cloudquant.com/datasource-config/local-file
Configure local or mounted file datasources for CloudQuant Data Liberator
# Local file (CSV/TSV)
Local file datasources read CSV, TSV, or other delimited flat files from a directory on the CloudQuant Data Liberator server or a mounted filesystem. This is the simplest file-based connection type and serves as the foundation for understanding all other file-based sources.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest on this connection, including Parquet, Arrow, Excel, XML, HDF5, PSV (2.1+), and PCAP (2.2+).
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | ---------------------------------------------------- |
| `connection_type` | string | Must be `"file"` |
| `behavior` | string | Must be `"file"` |
| `location` | string | Absolute path to the directory containing data files |
The `location` field should point to a **directory**, not an individual file. CloudQuant Data Liberator will scan the directory for files matching the `file_pattern` in `data_args`.
### Example connection
```json theme={null}
{
"name": "local-trades-connection",
"connection_type": "file",
"behavior": "file",
"location": "/data/trades"
}
```
## Dataset configuration (data\_args)
All file-based datasources share the same `data_args` fields. These control how CloudQuant Data Liberator finds, parses, and interprets your files.
### Required fields
| Field | Type | Description |
| ----------------- | -------------- | ---------------------------------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern to match files, e.g., `"*.csv"`, `"prefix_*.tsv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format string, or special values: `"muts"`, `"uts"`, `"nuts"`, `"datetime"`, `"date"` |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------------------------------------------------- |
| `sep_override` | string | `","` | Delimiter character: `","` (comma), `"\t"` (tab), `"\|"` (pipe), `";"` (semicolon) |
| `encoding` | string | `"utf-8"` | File encoding (e.g., `"utf-8"`, `"latin-1"`, `"ascii"`) |
| `data_dt_timezone` | string | `"UTC"` | Timezone of source data, e.g., `"UTC"`, `"America/New_York"` |
| `fname_dt_regex` | string | | Regex to extract a date from the filename |
| `fname_dt_format` | string | | strptime format for the date extracted by `fname_dt_regex` |
| `fname_dt_timezone` | string | | Timezone of the filename-derived date |
| `fname_dt_nudge` | int | `0` | Microsecond offset applied to filename-derived dates |
| `fname_dt_approx_seconds` | int | | Approximate number of seconds of data per file (used for query optimization) |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order for the resulting Arrow table |
| `arrow_timestamp` | bool | `true` | Whether to generate the human-readable `timestamp` column |
Set `fname_dt_approx_seconds` to `86400` for daily files. This helps CloudQuant Data Liberator skip files outside the query's time range, significantly improving performance for large directories.
## Complete example
Below is a full configuration showing both the connection and a dataset for daily trade CSV files.
### Connection
```json theme={null}
{
"name": "local-daily-trades",
"connection_type": "file",
"behavior": "file",
"location": "/data/daily-trades"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "local-daily-trades",
"data_args": {
"file_pattern": "trades_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "trade_time",
"data_dt_format": "%Y-%m-%d %H:%M:%S",
"data_dt_timezone": "America/New_York",
"data_key_column": "symbol",
"fname_dt_regex": "trades_(\\d{4}-\\d{2}-\\d{2})\\.csv",
"fname_dt_format": "%Y-%m-%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" }
]
}
```
Ensure the CloudQuant Data Liberator process has read permissions on the `location` directory and all files within it. Permission errors will cause silent failures during query execution.
## Tab-separated files (TSV)
For TSV files, set `sep_override` to `"\t"`:
```json theme={null}
{
"data_args": {
"file_pattern": "*.tsv",
"sep_override": "\t",
"data_dt_column": "date",
"data_dt_format": "%Y%m%d",
"data_dt_timezone": "UTC",
"data_key_column": "ticker"
}
}
```
## Composite key example
When the symbol is constructed from multiple columns:
```json theme={null}
{
"data_key_column": [
{ "type": "column", "value": "exchange" },
{ "type": "literal", "value": "_" },
{ "type": "column", "value": "ticker" }
]
}
```
This produces keys like `NYSE_AAPL`, `NASDAQ_MSFT`, etc.
## Multiple datetime columns
When the date and time are in separate columns:
```json theme={null}
{
"data_dt_column": ["trade_date", "trade_time"],
"data_dt_format": ["%Y-%m-%d", "%H:%M:%S.%f"]
}
```
CloudQuant Data Liberator concatenates the columns with a space before parsing, so the effective format becomes `"%Y-%m-%d %H:%M:%S.%f"`.
# SQL Server (MSSQL)
Source: https://knowledge.cloudquant.com/datasource-config/mssql
Configure Microsoft SQL Server datasources with ODBC support
# SQL Server (MSSQL)
SQL Server datasources use the ODBC driver and Microsoft ODBC Driver 18 for SQL Server.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on SQL Server connections.
## Connection configuration
### URI format
```
mssql://user:password@host:1433/database
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | ------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"mssql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | SQL Server connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (SQL Server default `1433`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Optional fields
| Field | Type | Default | Description |
| ------------ | ------ | ------- | ------------------------------------------------------- |
| `encryption` | string | | TLS encryption mode (e.g., `"yes"`, `"no"`, `"strict"`) |
### Environment variables
| Variable | Description |
| ------------------- | ----------------------------------------------------------------------------------------------- |
| `ODBC_MSSQL_DRIVER` | Override the ODBC driver name (default: `"ODBC Driver 18 for SQL Server"`) |
| `GODEBUG` | Set to `x509negativeserial=1` for servers using older certificates with negative serial numbers |
The `ODBC_MSSQL_DRIVER` environment variable allows you to use a different version of the Microsoft ODBC driver (e.g., Driver 17) without modifying the connection configuration.
### Example connection
```json theme={null}
{
"name": "mssql-market-data",
"connection_type": "mssql",
"connection_sub_type": "database",
"uri": "mssql://libuser:s3cureP@ss@db.example.com:1433/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss",
"encryption": "yes"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "mssql-trades",
"connection_type": "mssql",
"connection_sub_type": "database",
"uri": "mssql://libuser:s3cureP@ss@sqlserver.example.com:1433/trades_db",
"server": "sqlserver.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss",
"encryption": "yes"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "mssql-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "mssql-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native SQL Server `datetime2` or `datetimeoffset` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## Encryption configuration
SQL Server connections support several TLS encryption modes via the `encryption` field:
| Value | Description |
| ---------- | --------------------------------------------- |
| `"yes"` | Require encrypted connection (recommended) |
| `"no"` | Do not encrypt the connection |
| `"strict"` | Strict TLS 1.2+ encryption (SQL Server 2022+) |
When using `encryption: "yes"` with self-signed certificates, you may need to add `TrustServerCertificate=yes` to the connection. For production, always use properly signed certificates.
## Legacy certificate support
For SQL Server instances using older certificates with negative serial numbers:
```bash theme={null}
export GODEBUG=x509negativeserial=1
```
This is common with older SQL Server installations that have not rotated their TLS certificates.
## Driver and dependencies
| Component | Details |
| ---------------- | ------------------------------------------------ |
| Driver | `ODBC driver` with ODBC Driver 18 for SQL Server |
| Protocol | ODBC / TDS |
| Arrow conversion | High-performance ODBC driver |
| ODBC required | Yes (Microsoft ODBC Driver 18) |
| Default port | 1433 |
The Microsoft ODBC Driver 18 must be installed on the CloudQuant Data Liberator host. To use a different driver version, set the `ODBC_MSSQL_DRIVER` environment variable.
## Troubleshooting
### ODBC driver not found
Verify the Microsoft ODBC Driver 18 is installed and registered:
```bash theme={null}
# List installed ODBC drivers
odbcinst -q -d
# Expected output should include:
# [ODBC Driver 18 for SQL Server]
```
Install the driver if missing:
```bash Debian/Ubuntu theme={null}
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/22.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update && ACCEPT_EULA=Y apt-get install -y msodbcsql18
```
```bash RHEL/CentOS theme={null}
curl https://packages.microsoft.com/config/rhel/8/prod.repo > /etc/yum.repos.d/mssql-release.repo
ACCEPT_EULA=Y yum install -y msodbcsql18
```
### Using a different driver version
Override the default driver with an environment variable:
```bash theme={null}
export ODBC_MSSQL_DRIVER="ODBC Driver 17 for SQL Server"
```
### Certificate errors (x509)
If you see `x509: certificate signed by unknown authority` or negative serial number errors:
```bash theme={null}
export GODEBUG=x509negativeserial=1
```
### Connection timeout
SQL Server may be configured to listen on a non-default port or require a named instance. Verify connectivity:
```bash theme={null}
# Test port connectivity
nc -zv sqlserver.example.com 1433
# For named instances, use the SQL Browser port (1434/UDP) or specify the port directly
```
### Windows authentication
CloudQuant Data Liberator uses SQL Server authentication (username/password). Windows/Kerberos authentication is not supported through this connection type.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed.
* For partitioned tables, use `dt_regex` to scope queries to relevant partitions.
* Consider using `datetime2` over `datetime` for better precision and performance.
# MySQL
Source: https://knowledge.cloudquant.com/datasource-config/mysql
Configure MySQL datasources with ODBC support
# MySQL
MySQL datasources use the ODBC driver and the MariaDB ODBC connector. The connection string is built internally from the provided fields.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on MySQL connections.
## Connection configuration
### URI format
```
mysql://user:password@host:3306/database
```
The URI is used as a reference identifier. CloudQuant Data Liberator builds the actual ODBC connection string internally from the individual connection fields (`server`, `database`, `username`, `password`).
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | -------------------------------------------------------------------- |
| `connection_type` | string | Must be `"mysql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | MySQL connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (MySQL default `3306`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Optional fields
| Field | Type | Default | Description |
| -------------- | ------ | --------- | ------------------------------------------------------------ |
| `ssl_disabled` | string | `"False"` | Set to `"True"` to disable SSL. Values: `"True"` / `"False"` |
### Example connection
```json theme={null}
{
"name": "mysql-market-data",
"connection_type": "mysql",
"connection_sub_type": "database",
"uri": "mysql://libuser:s3cureP@ss@db.example.com:3306/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss",
"ssl_disabled": "False"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "mysql-trades",
"connection_type": "mysql",
"connection_sub_type": "database",
"uri": "mysql://libuser:s3cureP@ss@db.example.com:3306/trades_db",
"server": "db.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss",
"ssl_disabled": "False"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "mysql-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "mysql-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native MySQL `DATETIME` or `TIMESTAMP` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## SSL configuration
By default, SSL is enabled. To disable SSL (e.g., for local development):
```json theme={null}
{
"ssl_disabled": "True"
}
```
Disabling SSL in production is not recommended. Always use SSL when connecting over untrusted networks. The MariaDB ODBC driver supports TLS 1.2+ by default.
## Driver and dependencies
| Component | Details |
| ---------------- | ------------------------------------ |
| Driver | `ODBC driver` with MariaDB ODBC 3.2+ |
| Protocol | ODBC |
| Arrow conversion | High-performance ODBC driver |
| ODBC required | Yes (MariaDB ODBC connector) |
| Default port | 3306 |
The MariaDB ODBC 3.2+ driver must be installed on the CloudQuant Data Liberator host. This driver is compatible with both MySQL and MariaDB servers. Verify installation with `odbcinst -q -d`.
## Troubleshooting
### ODBC driver not found
If you see errors about missing ODBC drivers, verify the MariaDB ODBC driver is installed and registered:
```bash theme={null}
# List installed ODBC drivers
odbcinst -q -d
# Expected output should include:
# [MariaDB ODBC 3.2 Driver]
```
Install the driver if missing:
```bash Debian/Ubuntu theme={null}
apt-get install libmariadb3 odbc-mariadb
```
```bash RHEL/CentOS theme={null}
yum install MariaDB-connector-odbc
```
### Connection refused
Verify the MySQL server is accepting connections on the specified host and port. Check that the `bind-address` in the MySQL configuration allows remote connections.
```bash theme={null}
mysql -h db.example.com -P 3306 -u libuser -p marketdata
```
### Authentication failed
Ensure the user has been granted access from the CloudQuant Data Liberator host's IP address:
```sql theme={null}
GRANT SELECT ON trades_db.* TO 'libuser'@'liberator-host' IDENTIFIED BY 's3cureP@ss';
FLUSH PRIVILEGES;
```
### SSL handshake errors
If SSL connections fail, check that the MySQL server's SSL certificate is valid and trusted. Set `ssl_disabled` to `"True"` temporarily to confirm the issue is SSL-related.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in MySQL.
* For partitioned tables, ensure `dt_regex` is correctly scoping queries to relevant partitions.
# Oracle
Source: https://knowledge.cloudquant.com/datasource-config/oracle
Configure Oracle Database datasources with Oracle database driver thin mode
# Oracle
Oracle datasources use the `Oracle database driver` in thin mode. No Oracle Instant Client installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on Oracle connections.
## Connection configuration
### URI format
```
oracle://user:password@host:1521/service_name
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | --------------------------------------------------------------------- |
| `connection_type` | string | Must be `"oracle"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | Oracle connection string |
| `server` | string | Database host address |
| `database` | string | Oracle service name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (Oracle default `1521`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`). Missing tables return a clear error instead of an empty sample.
### Optional fields
| Field | Type | Default | Description |
| ----------- | ------ | ------- | --------------------------------------------- |
| `auth_mode` | string | | Oracle authentication mode (e.g., `"SYSDBA"`) |
| `schema` | string | | Oracle schema name to query against |
### Environment variables
| Variable | Description |
| ----------------------- | ----------------------------------------------- |
| `ORACLE_DEBUG_BOUNDARY` | Enable debug timing for Oracle query boundaries |
### Example connection
```json theme={null}
{
"name": "oracle-market-data",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://libuser:s3cureP@ss@db.example.com:1521/ORCL",
"server": "db.example.com",
"database": "ORCL",
"username": "libuser",
"password": "s3cureP@ss"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database/service name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "oracle-trades",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://libuser:s3cureP@ss@oradb.example.com:1521/TRADEDB",
"server": "oradb.example.com",
"database": "TRADEDB",
"username": "libuser",
"password": "s3cureP@ss",
"schema": "MARKET_DATA"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "oracle-trades",
"data_args": {
"table_options": [
{
"db": "TRADEDB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "oracle-trades",
"batch_size": 10000,
"data_dt_column": "TRADE_TIME",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "SYMBOL"
},
"schema": [
{ "name": "SYMBOL", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "TRADE_TIME", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "PRICE", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "VOLUME", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "EXCHANGE", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Oracle column names are uppercase by default. Use uppercase names in your schema definition and `data_args` fields to match Oracle's metadata.
## Authentication modes
The `auth_mode` field supports Oracle authentication modes for privileged connections:
```json theme={null}
{
"name": "oracle-admin",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://sys:adminP@ss@db.example.com:1521/ORCL",
"server": "db.example.com",
"database": "ORCL",
"username": "sys",
"password": "adminP@ss",
"auth_mode": "SYSDBA"
}
```
Using `SYSDBA` authentication in production datasource connections is not recommended. Create a dedicated read-only user with `SELECT` privileges on the required tables.
## Schema-qualified tables
When the `schema` field is set on the connection, CloudQuant Data Liberator queries tables within that schema. This is useful when the database user's default schema differs from where the data resides:
```json theme={null}
{
"schema": "MARKET_DATA",
"table_options": [
{
"db": "TRADEDB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
]
}
```
This queries `MARKET_DATA.DAILY_TRADES` rather than relying on the user's default schema.
## Microsecond timestamp precision
Oracle `TIMESTAMP(6)` columns support microsecond precision. A custom-patched `oracledb 3.4.0` is available for CloudQuant Data Liberator that preserves full microsecond precision during data extraction.
The standard `Oracle database driver 3.3+` driver supports microsecond timestamps. The custom-patched 3.4.0 build provides additional optimizations for high-frequency timestamp extraction. Contact your CloudQuant Data Liberator administrator for the patched driver.
## Driver and dependencies
| Component | Details |
| ---------------------- | ----------------------------------------- |
| Driver | `Oracle database driver` 3.3+ (thin mode) |
| Protocol | Oracle Net (TNS) |
| Arrow conversion | Via Oracle driver native conversion |
| Oracle Client required | No (thin mode) |
| Default port | 1521 |
Thin mode connects directly to Oracle without requiring Oracle Instant Client. This simplifies deployment and eliminates native library dependencies.
## Troubleshooting
### Connection refused
Verify the Oracle listener is running and accepting connections on the specified host and port:
```bash theme={null}
# Test port connectivity
nc -zv oradb.example.com 1521
# Test with sqlplus (if available)
sqlplus libuser/s3cureP@ss@oradb.example.com:1521/TRADEDB
```
### ORA-12514: TNS listener does not currently know of service
The service name in the URI does not match any registered service on the Oracle listener. Verify the service name:
```sql theme={null}
-- On the Oracle server
SELECT name FROM v$services;
```
### ORA-01017: invalid username/password
Ensure the username and password are correct. Oracle passwords are case-sensitive (depending on the `SEC_CASE_SENSITIVE_LOGON` parameter).
### Table not found (ORA-00942)
Check that the user has `SELECT` privileges on the target table and that the schema is correctly specified:
```sql theme={null}
GRANT SELECT ON MARKET_DATA.DAILY_TRADES TO libuser;
```
### Debug timing
To enable debug timing for Oracle query boundaries, set the environment variable:
```bash theme={null}
export ORACLE_DEBUG_BOUNDARY=1
```
This logs timing information for each query phase, useful for diagnosing performance bottlenecks.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in Oracle.
* For partitioned tables, verify that Oracle partition pruning is being used by checking the execution plan.
* Use `TIMESTAMP(6)` columns for microsecond precision rather than `DATE` (which only supports second precision).
# Datasource Configuration Overview
Source: https://knowledge.cloudquant.com/datasource-config/overview
Guide to configuring datasource connections in CloudQuant Data Liberator
# Datasource configuration
CloudQuant Data Liberator supports a wide range of datasource types for ingesting time series data. Each datasource requires a **connection** (how to reach the data) and a **dataset** (what data to extract and how to interpret it).
See [Supported Data Formats](/datasource-config/supported-formats) for the canonical list of file extensions and data source categories Liberator can ingest, including formats added after 2.0.
## Supported datasource types
### File-based sources
| Type | Description |
| --------------------------------------------------------------- | ------------------------------------------------------------------- |
| [Local File (CSV/TSV)](/datasource-config/local-file) | Flat files on local/mounted storage |
| [S3](/datasource-config/s3) | Amazon S3 or S3-compatible object storage |
| [Azure Blob Storage](/datasource-config/azure-blob) | Microsoft Azure Blob containers |
| [SharePoint / OneDrive](/datasource-config/sharepoint-onedrive) | Microsoft 365 file storage via share link (2.2+); wildcards in 2.3+ |
| [SFTP](/datasource-config/sftp) | SSH File Transfer Protocol servers |
| [FTPS](/datasource-config/ftps) | FTP over TLS/SSL |
| [CIFS/SMB](/datasource-config/cifs) | Windows/Samba network file shares |
### Database sources
| Type | Description |
| ------------------------------------------- | -------------------------------------- |
| [PostgreSQL](/datasource-config/postgresql) | High-performance native driver |
| [MySQL](/datasource-config/mysql) | Via ODBC driver (MySQL-compatible) |
| [SQL Server](/datasource-config/mssql) | Via ODBC driver (ODBC Driver 18) |
| [Oracle](/datasource-config/oracle) | Via Oracle database driver (thin mode) |
| [Snowflake](/datasource-config/snowflake) | High-performance native driver |
### API sources
| Type | Description |
| ------------------------------------- | ---------------------------------- |
| [HubSpot](/datasource-config/hubspot) | HubSpot CRM objects via API (2.3+) |
### File and data formats
Beyond the delimited text examples in each connection guide, Liberator also ingests Parquet, Arrow IPC, Excel, XML, HDF5, ZIP archives, database tables, and API payloads. PSV (2.1+) and PCAP/PCAPng FIX capture (2.2+) are documented in [Supported Data Formats](/datasource-config/supported-formats).
## Architecture: connection + dataset
Every datasource in CloudQuant Data Liberator is composed of two parts:
### Connection
Defines **how to reach** the data — credentials, endpoints, paths, and transport protocol.
```
Connection → "Where is the data and how do I authenticate?"
```
### Dataset
Defines **what to extract** — which table/files, timestamp columns, key columns, schema, and data frequency.
```
Dataset → "What data do I want and how do I interpret it?"
```
In Liberator **2.3** and later, you can create both through the **AI Assisted** Create Dataset flow. Describe the source in plain language (or paste connection details), then review the auto-filled wizard steps before creating. See [AI-driven dataset onboarding](/administration/ai-dataset-onboarding).
## Common configuration concepts
### Timestamp configuration
All datasources require timestamp configuration to map source data into CloudQuant Data Liberator's microsecond timestamp (`muts`) format:
| Field | Description |
| ------------------ | ----------------------------------------------------------------- |
| `data_dt_column` | Column(s) containing the datetime |
| `data_dt_format` | Format string or parsing specification |
| `data_dt_timezone` | Timezone of the source data (e.g., `"UTC"`, `"America/New_York"`) |
| `data_dt_nudge` | Microsecond offset applied to timestamps |
#### Supported datetime formats
| Format | Description |
| --------------------- | ---------------------------------------------- |
| `"%Y-%m-%d %H:%M:%S"` | Standard strptime format |
| `"datetime"` | Native database datetime column |
| `"date"` | Native date column (date32/date64) |
| `"muts"` | Unix epoch microseconds |
| `"uts"` | Unix epoch seconds |
| `"nuts"` | Unix epoch nanoseconds |
| `true` | Auto-detect native datetime (database sources) |
### Key column configuration
The `data_key_column` field defines the symbol/key used for filtering queries:
```python theme={null}
# Simple string
"data_key_column": "symbol"
# Composite key with literals and columns
"data_key_column": [
{"type": "column", "value": "exchange"},
{"type": "literal", "value": "_"},
{"type": "column", "value": "ticker"}
]
```
### Schema definition
Each column in a dataset schema requires:
```json theme={null}
{
"name": "column_name",
"type": "int64",
"group": "value",
"description": "Human-readable description",
"display_name": "Display Name"
}
```
**Column types:** `string`, `int64`, `uint64`, `double`, `float`, `bool`, `date32`, `date64`, `time64`
**Column groups:**
* `key` — Symbol/key columns
* `time` — Timestamp columns
* `value` — Data columns
* `meta` — System columns (`_seq`, `muts`, etc.)
### Auto-generated columns
CloudQuant Data Liberator automatically generates these columns if not present in source data:
| Column | Type | Description |
| ----------- | ------ | -------------------------------------------- |
| `_seq` | uint64 | Sequential row number within partition |
| `muts` | int64 | Microseconds since Unix epoch |
| `timestamp` | string | Human-readable timestamp (America/New\_York) |
| `symbol` | string | Key column (copied from `data_key_column`) |
### File name date extraction
For file-based sources, dates can be extracted from filenames:
| Field | Description | Example |
| ------------------------- | --------------------------------------- | ------------------------------- |
| `fname_dt_regex` | Regex to match date portion of filename | `data_(\d{4}-\d{2}-\d{2})\.csv` |
| `fname_dt_format` | strptime format for the matched portion | `%Y-%m-%d` |
| `fname_dt_timezone` | Timezone of the filename date | `UTC` |
| `fname_dt_nudge` | Microsecond offset | `0` |
| `fname_dt_approx_seconds` | Approximate seconds per file | `86400` |
# PostgreSQL
Source: https://knowledge.cloudquant.com/datasource-config/postgresql
Configure PostgreSQL datasources with high-performance native driver support
# PostgreSQL
PostgreSQL datasources use a high-performance native driver for optimized data transfer. No ODBC installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on PostgreSQL connections.
## Connection configuration
### URI format
```
postgresql://user:password@host:5432/database
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | ------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"postgresql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | PostgreSQL connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (PostgreSQL default `5432`) |
The native driver communicates with the database using the PostgreSQL binary protocol and produces data batches directly, avoiding row-by-row conversion overhead. In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Example connection
```json theme={null}
{
"name": "postgres-market-data",
"connection_type": "postgresql",
"connection_sub_type": "database",
"uri": "postgresql://libuser:s3cureP@ss@db.example.com:5432/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "postgres-trades",
"connection_type": "postgresql",
"connection_sub_type": "database",
"uri": "postgresql://libuser:s3cureP@ss@db.example.com:5432/trades_db",
"server": "db.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "postgres-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "postgres-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native PostgreSQL `timestamp` or `timestamptz` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## Partitioned table example
For tables partitioned by date (e.g., `trades_2024_01`, `trades_2024_02`):
```json theme={null}
{
"table_options": [
{
"db": "trades_db",
"table_like": "trades_%",
"token": "trades",
"zone": "America/New_York",
"dt_regex": "trades_(\\d{4})_(\\d{2})",
"dt_format": "%Y_%m"
}
]
}
```
## Driver and dependencies
| Component | Details |
| --------------- | -------------------------- |
| Driver | PostgreSQL native driver |
| Protocol | PostgreSQL binary protocol |
| Data conversion | Native (high-performance) |
| ODBC required | No |
| Default port | 5432 |
The native driver is bundled with CloudQuant Data Liberator. No additional driver installation is needed on the host system.
## Troubleshooting
### Connection refused
Verify the PostgreSQL server is accepting connections on the specified host and port. Check `pg_hba.conf` for client authentication rules.
```bash theme={null}
psql -h db.example.com -p 5432 -U libuser -d marketdata
```
### Authentication failed
Ensure the `username` and `password` in the connection match a valid PostgreSQL role. Check that the role has `CONNECT` privilege on the database and `SELECT` privilege on the target tables.
### SSL/TLS errors
If the server requires SSL, append `?sslmode=require` to the URI:
```
postgresql://libuser:s3cureP@ss@db.example.com:5432/marketdata?sslmode=require
```
Using `sslmode=disable` in production is not recommended. Always use `sslmode=require` or `sslmode=verify-full` when connecting over untrusted networks.
### Empty results from partitioned tables
Verify that `dt_regex` correctly matches the table name suffix and that `dt_format` corresponds to the captured groups. Test the regex against your actual table names.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in PostgreSQL.
* Use `table_like` to target specific tables rather than scanning all partitions.
# Amazon S3
Source: https://knowledge.cloudquant.com/datasource-config/s3
Configure S3 or S3-compatible object storage datasources
# Amazon S3
S3 datasources allow CloudQuant Data Liberator to read CSV, TSV, and Parquet files directly from Amazon S3 buckets or S3-compatible object storage services (MinIO, Wasabi, Backblaze B2, etc.).
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest on S3, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------------- | ------ | --------------------------------------------------- |
| `connection_type` | string | Must be `"s3"` |
| `aws_access_key_id` | string | AWS access key ID |
| `aws_secret_access_key` | string | AWS secret access key |
| `bucket` | string | S3 bucket name |
| `endpoint` | string | S3 endpoint URL, e.g., `"https://s3.amazonaws.com"` |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ------ | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `prefix` | string | `""` | Key prefix (virtual directory) within the bucket |
| `request_style` | string | auto (2.4+) | S3 addressing: `"path"` or `"virtual"`. In Liberator **2.4** and later, Test Connection probes the endpoint and persists the correct style. Override only if the probe is wrong. |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
| `object_key` | string | | Object key pattern for file selection |
In Liberator **2.4** and later, Test Connection auto-detects path-style vs virtual-hosted addressing for AWS and S3-compatible endpoints (MinIO, Ceph, Wasabi, Oracle Cloud, and custom hosts). You can still set `request_style` explicitly. For older releases, set `request_style` to `"path"` for most S3-compatible services and keep `"virtual"` for AWS S3.
### Example connection
```json theme={null}
{
"name": "s3-market-data",
"connection_type": "s3",
"aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
"aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"bucket": "my-market-data",
"endpoint": "https://s3.amazonaws.com",
"prefix": "daily/equities/",
"request_style": "virtual"
}
```
Never commit AWS credentials to version control. Use environment variables or a secrets manager to inject credentials at deployment time.
## CSV/TSV dataset
The `data_args` are identical to [Local File](/datasource-config/local-file) sources. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
### Complete CSV example
```json theme={null}
{
"name": "s3-equity-trades",
"connection": "s3-market-data",
"data_args": {
"file_pattern": "trades_*.csv",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": "timestamp",
"data_dt_format": "%Y-%m-%d %H:%M:%S.%f",
"data_dt_timezone": "UTC",
"data_key_column": "symbol",
"fname_dt_regex": "trades_(\\d{8})\\.csv",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "UTC",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "timestamp", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "size", "type": "int64", "group": "value", "description": "Trade size" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
## Parquet dataset
This provides passthrough access to Parquet data without intermediate caching, leveraging Arrow's native Parquet reader.
Parquet files offer zero-copy reads and columnar pushdown. This is significantly more efficient than converting Parquet to CSV.
### Complete Parquet example
```json theme={null}
{
"name": "s3-equity-bars-parquet",
"connection": "s3-market-data",
"data_args": {
"file_pattern": "bars_*.parquet",
"data_dt_column": "bar_time",
"data_dt_format": "datetime",
"data_dt_timezone": "America/New_York",
"data_key_column": "symbol",
"fname_dt_regex": "bars_(\\d{4}-\\d{2}-\\d{2})\\.parquet",
"fname_dt_format": "%Y-%m-%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "bar_time", "type": "string", "group": "time", "description": "Bar timestamp" },
{ "name": "open", "type": "double", "group": "value", "description": "Open price" },
{ "name": "high", "type": "double", "group": "value", "description": "High price" },
{ "name": "low", "type": "double", "group": "value", "description": "Low price" },
{ "name": "close", "type": "double", "group": "value", "description": "Close price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Bar volume" }
]
}
```
## S3-compatible storage
### MinIO example
```json theme={null}
{
"name": "minio-connection",
"connection_type": "s3",
"aws_access_key_id": "minio-access-key",
"aws_secret_access_key": "minio-secret-key",
"bucket": "market-data",
"endpoint": "https://minio.internal.example.com:9000",
"request_style": "path"
}
```
Most S3-compatible services require `request_style` set to `"path"`. Only AWS S3 defaults to virtual-hosted style.
## IAM permissions
The IAM user or role associated with the access key needs at minimum:
* `s3:GetObject` on the bucket objects
* `s3:ListBucket` on the bucket
See the [S3 Bucket Setup](/integrations/s3-bucket-setup) guide for detailed IAM policy configuration.
# SFTP
Source: https://knowledge.cloudquant.com/datasource-config/sftp
Configure SFTP datasources
# SFTP
SFTP (SSH File Transfer Protocol) datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from remote servers over an encrypted SSH connection. CloudQuant Data Liberator mounts the remote directory via SSHFS/FUSE.
See [Supported Data Formats](/datasource-config/supported-formats) for every file extension Liberator can ingest over SFTP, including formats added in 2.1 and 2.2.
## Connection configuration
### Required fields
| Field | Type | Description |
| ----------------- | ------ | ---------------------------------- |
| `connection_type` | string | Must be `"sftp"` |
| `host` | string | SFTP server hostname or IP address |
| `user` | string | Username for authentication |
You must provide either `password` or `key` for authentication. If both are specified, key-based authentication takes precedence.
### Optional fields
| Field | Type | Default | Description |
| ------------- | ------ | ------- | ----------------------------------------------------------------- |
| `port` | int | `22` | SSH port number |
| `password` | string | | Password for password-based authentication |
| `key` | string | | SSH private key content (PEM format) for key-based authentication |
| `prefix` | string | `""` | Remote directory path to use as root |
| `mount_point` | string | | Local mount path for FUSE-based access |
| `config_name` | string | | Internal configuration identifier |
### Example connection (password authentication)
```json theme={null}
{
"name": "sftp-vendor-data",
"connection_type": "sftp",
"host": "sftp.vendor.example.com",
"port": 22,
"user": "datauser",
"password": "s3cur3P@ssw0rd",
"prefix": "/data/daily-feeds/"
}
```
### Example connection (key authentication)
```json theme={null}
{
"name": "sftp-internal-data",
"connection_type": "sftp",
"host": "data-server.internal.net",
"port": 2222,
"user": "liberator-svc",
"key": "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNza...\n-----END OPENSSH PRIVATE KEY-----",
"prefix": "/exports/market-data/"
}
```
Avoid embedding private keys or passwords directly in configuration files. Use environment variables or a secrets manager to inject credentials at deployment time.
## Dataset configuration (data\_args)
The `data_args` fields are identical to all file-based sources. See [Local File](/datasource-config/local-file) for the full reference. The `file_pattern` is evaluated relative to the `prefix` configured on the connection.
### Required data\_args
| Field | Type | Description |
| ----------------- | -------------- | ------------------------------------------------------------------- |
| `file_pattern` | string | Glob pattern relative to the prefix, e.g., `"*.csv"` |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | string or list | strptime format or special values (`"muts"`, `"uts"`, `"datetime"`) |
| `data_key_column` | string or list | Symbol/key column(s) |
### Optional data\_args
| Field | Type | Default | Description |
| ------------------------- | ------ | -------------------- | ---------------------------------------- |
| `sep_override` | string | `","` | Delimiter character |
| `encoding` | string | `"utf-8"` | File encoding |
| `data_dt_timezone` | string | `"UTC"` | Source data timezone |
| `fname_dt_regex` | string | | Regex to extract date from filename |
| `fname_dt_format` | string | | strptime format for filename date |
| `fname_dt_timezone` | string | | Timezone of filename date |
| `fname_dt_nudge` | int | `0` | Microsecond offset for filename date |
| `fname_dt_approx_seconds` | int | | Approximate seconds per file |
| `arrow_sort` | list | `["symbol", "muts"]` | Sort order |
| `arrow_timestamp` | bool | `true` | Generate human-readable timestamp column |
## Complete example
### Connection
```json theme={null}
{
"name": "sftp-trades-feed",
"connection_type": "sftp",
"host": "sftp.dataprovider.com",
"port": 22,
"user": "cq-ingest",
"password": "vendorPassword123",
"prefix": "/feeds/trades/"
}
```
### Dataset
```json theme={null}
{
"name": "vendor-trades",
"connection": "sftp-trades-feed",
"data_args": {
"file_pattern": "trades_*.csv.gz",
"sep_override": ",",
"encoding": "utf-8",
"data_dt_column": ["date", "time"],
"data_dt_format": ["%Y%m%d", "%H:%M:%S.%f"],
"data_dt_timezone": "America/New_York",
"data_key_column": "symbol",
"fname_dt_regex": "trades_(\\d{8})\\.csv\\.gz",
"fname_dt_format": "%Y%m%d",
"fname_dt_timezone": "America/New_York",
"fname_dt_approx_seconds": 86400,
"arrow_sort": ["symbol", "muts"],
"arrow_timestamp": true
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "date", "type": "string", "group": "time", "description": "Trade date" },
{ "name": "time", "type": "string", "group": "time", "description": "Trade time" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "size", "type": "int64", "group": "value", "description": "Trade size" },
{ "name": "condition", "type": "string", "group": "value", "description": "Sale condition code" }
]
}
```
CloudQuant Data Liberator supports reading gzip-compressed files (`.csv.gz`) transparently. Use compressed files on SFTP connections to reduce transfer time over slow or high-latency links.
## Network requirements
Ensure the following network connectivity from the CloudQuant Data Liberator host:
| Requirement | Detail |
| ------------------ | ------------------------------------------------------------------ |
| **Outbound port** | TCP port 22 (or custom port) to the SFTP server |
| **DNS resolution** | The hostname must resolve from the CloudQuant Data Liberator host |
| **Firewall rules** | Whitelist the CloudQuant Data Liberator host IP on the SFTP server |
| **SSH host key** | The server's host key must be trusted (added to known\_hosts) |
# SharePoint and OneDrive
Source: https://knowledge.cloudquant.com/datasource-config/sharepoint-onedrive
Connect SharePoint or OneDrive data sources using a Microsoft share link
# SharePoint and OneDrive
SharePoint and OneDrive connections let Liberator read delimited files, Parquet, Excel, and other [supported formats](/datasource-config/supported-formats) from Microsoft 365 storage.
In Liberator 2.2 and later, paste a standard **share link** copied from the browser or desktop to connect. In Liberator **2.3** and later, wildcard folder and file patterns (for example, `reports/2026-*/trades_*.csv`) are supported for both SharePoint and OneDrive sources.
## Connect with a share link
In the browser or desktop app, open the file or folder, then **Share → Copy link**. The link looks like `https://your-org.sharepoint.com/:f:/s/...`.
Go to **Connections → Add Connection** and select **SharePoint** or **OneDrive**.
Paste the link into the **Share link** field. Liberator clears any previously entered folder path when a share link is detected.
Complete the **Sign in with Microsoft** dialog. If it does not appear automatically, click **Sign in with Microsoft**.
Click **Test Connection**. On success, the resolved internal path replaces the share link in the field. Confirm the account has at least read access to the linked location.
Continue through dataset configuration (file pattern, columns, date/time) as for other file-based sources.
## Fixed date and time (non-time-series files)
For static reference files without a date column:
1. On the **Date / Time** step, select **Fixed date and time**.
2. Enter the timestamp Liberator should assign to records.
If the source has a date column, select it from the **Date column** dropdown instead.
## Date format picker (2.2+)
When choosing a date format during onboarding, Liberator:
* Shows a **live example from your sample data** (for example, `25 Jun 2026` instead of generic `DD MMM YYYY`).
* **Ranks formats by likelihood**, with the best match at the top.
Select the example that matches your source dates.
## Related
File extensions available on SharePoint and OneDrive connections
Shared `data_args` fields for file-based datasets
# Snowflake
Source: https://knowledge.cloudquant.com/datasource-config/snowflake
Configure Snowflake datasources with native high-performance driver
# Snowflake
Snowflake datasources use the Snowflake native driver for high-performance native data transfer. No ODBC installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on Snowflake connections.
## Connection configuration
### URI format
```
snowflake://user:password@account/database/schema?warehouse=compute_wh&role=accountadmin
```
### Required fields
| Field | Type | Description |
| --------------------- | ------ | --------------------------------------------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"snowflake"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | Snowflake connection string |
| `server` | string | Snowflake account URL (e.g., `"xy12345.us-east-1.snowflakecomputing.com"`) |
| `database` | string | Snowflake database name |
| `username` | string | Snowflake user |
| `password` | string | Snowflake password. In Liberator **2.4+**, special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`) are supported |
| `account` | string | Snowflake account identifier (e.g., `"xy12345.us-east-1"`) |
| `warehouse` | string | Compute warehouse name |
| `role` | string | Snowflake role |
### Optional fields
| Field | Type | Default | Description |
| -------- | ------ | ------- | --------------------- |
| `schema` | string | | Snowflake schema name |
### Environment variables
| Variable | Description |
| --------------------- | ------------------------------------------------------ |
| `SNOWFLAKE_DEBUGGING` | Enable verbose debug logging for Snowflake connections |
### Example connection
```json theme={null}
{
"name": "snowflake-market-data",
"connection_type": "snowflake",
"connection_sub_type": "database",
"uri": "snowflake://libuser:s3cureP@ss@xy12345.us-east-1/MARKETDATA/PUBLIC?warehouse=COMPUTE_WH&role=DATA_READER",
"server": "xy12345.us-east-1.snowflakecomputing.com",
"database": "MARKETDATA",
"username": "libuser",
"password": "s3cureP@ss",
"account": "xy12345.us-east-1",
"warehouse": "COMPUTE_WH",
"role": "DATA_READER",
"schema": "PUBLIC"
}
```
The Snowflake native driver uses Snowflake's native result set format. Data is returned as batches directly from the Snowflake service with no row-by-row conversion.
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "snowflake-trades",
"connection_type": "snowflake",
"connection_sub_type": "database",
"uri": "snowflake://libuser:s3cureP@ss@xy12345.us-east-1/TRADES_DB/MARKET?warehouse=COMPUTE_WH&role=accountadmin",
"server": "xy12345.us-east-1.snowflakecomputing.com",
"database": "TRADES_DB",
"username": "libuser",
"password": "s3cureP@ss",
"account": "xy12345.us-east-1",
"warehouse": "COMPUTE_WH",
"role": "accountadmin",
"schema": "MARKET"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "snowflake-trades",
"data_args": {
"table_options": [
{
"db": "TRADES_DB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "snowflake-trades",
"batch_size": 10000,
"data_dt_column": "TRADE_TIME",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "SYMBOL"
},
"schema": [
{ "name": "SYMBOL", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "TRADE_TIME", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "PRICE", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "VOLUME", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "EXCHANGE", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Snowflake identifiers are uppercase by default. Use uppercase names in your schema definition and `data_args` fields to match Snowflake's metadata, unless the objects were created with double-quoted lowercase names.
## Warehouse and role configuration
The `warehouse` and `role` fields control compute resources and access permissions:
### Warehouse
The warehouse determines the compute cluster used to execute queries. Choose a warehouse sized appropriately for your query workload:
```json theme={null}
{
"warehouse": "COMPUTE_WH"
}
```
Ensure the specified warehouse is set to auto-resume, or queries will fail when the warehouse is suspended. CloudQuant Data Liberator does not issue `ALTER WAREHOUSE ... RESUME` commands.
### Role
The role determines which database objects are accessible:
```json theme={null}
{
"role": "DATA_READER"
}
```
Create a dedicated read-only role for CloudQuant Data Liberator connections rather than using `accountadmin`. Grant `SELECT` on the required databases and schemas to this role.
## Debug logging
Enable verbose debug logging for Snowflake connections to diagnose connectivity or query issues:
```bash theme={null}
export SNOWFLAKE_DEBUGGING=1
```
This produces detailed logs including connection negotiation, query execution timing, and data batch transfer metrics.
## Driver and dependencies
| Component | Details |
| --------------- | --------------------------------------- |
| Driver | `Snowflake native driver` |
| Protocol | Snowflake REST API / streaming protocol |
| Data conversion | Native (high-performance) |
| ODBC required | No |
| Default port | 443 (HTTPS) |
The Snowflake native driver is bundled with CloudQuant Data Liberator. No additional driver installation is needed on the host system. All communication uses HTTPS on port 443.
## Troubleshooting
### Account identifier format
The `account` field must include the region if your account is not in the default AWS us-west-2 region:
```
# US East (Ohio)
xy12345.us-east-1
# EU (Frankfurt)
xy12345.eu-central-1
# Azure (East US 2)
xy12345.east-us-2.azure
```
### Authentication failed (390100)
Verify the username, password, and account identifier. Snowflake passwords are case-sensitive. If multi-factor authentication (MFA) is enabled for the user, you may need to use a service account without MFA.
### Warehouse is suspended
If queries fail with a warehouse suspension error, ensure the warehouse is configured with `AUTO_RESUME = TRUE`:
```sql theme={null}
ALTER WAREHOUSE COMPUTE_WH SET AUTO_RESUME = TRUE;
```
### Role does not have access
Verify the role has the required privileges:
```sql theme={null}
GRANT USAGE ON DATABASE TRADES_DB TO ROLE DATA_READER;
GRANT USAGE ON SCHEMA TRADES_DB.MARKET TO ROLE DATA_READER;
GRANT SELECT ON ALL TABLES IN SCHEMA TRADES_DB.MARKET TO ROLE DATA_READER;
```
### Network connectivity
Snowflake connections require HTTPS (port 443) access to `*.snowflakecomputing.com`. Ensure firewalls and proxy servers allow this traffic.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure Snowflake clustering keys align with your datetime and key columns.
* Use an appropriately sized warehouse (e.g., `MEDIUM` or `LARGE`) for large datasets.
* Check that the warehouse is not queued behind other workloads using Snowflake's query history.
# Supported Data Formats
Source: https://knowledge.cloudquant.com/datasource-config/supported-formats
File and data formats CloudQuant Data Liberator can ingest, by platform version
# Supported data formats
CloudQuant Data Liberator ingests time-series and tabular data through **connections** (where data lives) and **datasets** (how files or tables are interpreted). The tables below list every **file extension** and **data source category** the platform understands.
Formats are selected automatically from your `file_pattern`, file extension, or connection type when you onboard through the Liberator UI. You rarely need to set anything manually.
This page is the canonical format reference. Each [connection guide](/datasource-config/overview) links here for the formats available on that storage type.
## Supported in version 2.0
These formats were available in the Liberator **2.0** release and remain supported on current versions.
### Delimited text files
| Extension | Delimiter | Configuration |
| --------- | --------------- | ----------------------------------------------------- |
| `.csv` | Comma (default) | `file_pattern` such as `*.csv` |
| `.tsv` | Tab | `file_pattern` such as `*.tsv` |
| `.txt` | Auto-detected | Same pipeline as CSV; delimiter inferred when omitted |
Set `sep_override` in `data_args` to force comma, tab, pipe (`\|`), or semicolon when auto-detection is not sufficient.
### Columnar and binary files
| Extension | Description |
| -------------------- | ------------------------------------------------- |
| `.parquet` | Apache Parquet; columnar reads with type pushdown |
| `.arrow`, `.feather` | Apache Arrow IPC (Feather v2) |
On S3, Parquet datasets can use **passthrough** mode (direct read without intermediate caching). See the [S3 Parquet example](/datasource-config/s3#parquet-dataset).
### Structured office and scientific files
| Extension | Description | Extra configuration |
| --------------- | ------------------------- | ---------------------------------------------------- |
| `.xlsx`, `.xls` | Microsoft Excel workbooks | First sheet sampled at onboarding |
| `.xml` | XML documents | `xml_args` for element paths (set during onboarding) |
| `.h5`, `.hdf5` | HDF5 scientific arrays | `h5py_groups` for dataset path inside the file |
### Archives
| Extension | Description |
| --------- | --------------------------------------------------------------------------------------- |
| `.zip` | ZIP archive; Liberator inspects the inner file and applies the matching format delegate |
### Database sources
| Connection type | Engine | Access pattern |
| --------------- | ------------------------------ | ------------------------- |
| PostgreSQL | Native high-performance driver | Table or view per dataset |
| MySQL | ODBC (MariaDB-compatible) | Table or view per dataset |
| SQL Server | ODBC Driver 18 | Table or view per dataset |
| Oracle | Thin driver | Table or view per dataset |
| Snowflake | Native driver | Table or view per dataset |
### API-backed datasets
REST endpoints that return JSON tabular payloads can be onboarded as **API** connections. The platform normalizes responses into the same query surface as file- and database-backed datasets.
### Schema column types
Regardless of source format, dataset schemas use these column types:
`string`, `int64`, `uint64`, `double`, `float`, `bool`, `date32`, `date64`, `time64`
See [Datasource configuration overview](/datasource-config/overview#schema-definition) for column groups (`key`, `time`, `value`, `meta`).
## Formats added after version 2.0
The following ingest formats were added in subsequent releases. They are available on current CloudQuant-managed environments at those versions and later.
### Version 2.1 — PSV (pipe-separated values)
| Extension | Delimiter | Notes |
| --------- | ----------- | ----------------------------------------------- |
| `.psv` | Pipe (`\|`) | First-class extension alongside CSV and Parquet |
PSV files use the same onboarding flow as CSV: header row, per-column type inference, and configurable null sentinel. You can also ingest pipe-delimited `.csv` or `.txt` files by setting `sep_override` to `"\|"` without renaming the file.
### Version 2.2 — PCAP / PCAPng (FIX tick capture)
| Extension | Description |
| --------- | ------------------------------ |
| `.pcap` | Classic packet capture |
| `.pcapng` | Next-generation packet capture |
Liberator extracts **FIX-protocol messages** from TCP payloads in packet captures and exposes them through the standard query API. Typical columns include FIX tags such as `35` (MsgType), `49` (SenderCompID), `52` (SendingTime), and `55` (Symbol), plus `_pcap_ts_ns` for the capture timestamp.
Use PCAP datasets when you capture exchange feeds at the wire level and want the same query model as historical bar or trade datasets.
## Added in version 2.3
No new file or packet-capture ingest formats ship in 2.3. Connectivity additions in this release (HubSpot CRM objects, OneDrive/SharePoint wildcards) are covered in the [HubSpot](/datasource-config/hubspot) and [SharePoint and OneDrive](/datasource-config/sharepoint-onedrive) guides and in [What's New in Liberator 2.3](/whats-new/liberator-2.3).
## Added in version 2.4
No new file or packet-capture ingest formats ship in 2.4. Query, catalog, and connection changes in this release (symbol pattern filters, dataset health cards, S3 request-style auto-detect, required database port) are covered in [What's New in Liberator 2.4](/whats-new/liberator-2.4).
## Where each format applies
All **file-based** formats in the tables above can be stored on any file-backed connection type:
| Connection | Guide |
| ----------------------------- | ------------------------------------------- |
| Local / mounted directory | [Local File](/datasource-config/local-file) |
| Amazon S3 (and S3-compatible) | [S3](/datasource-config/s3) |
| Azure Blob Storage | [Azure Blob](/datasource-config/azure-blob) |
| SFTP | [SFTP](/datasource-config/sftp) |
| FTPS | [FTPS](/datasource-config/ftps) |
| CIFS / SMB | [CIFS](/datasource-config/cifs) |
**Database** and **API** formats map to their respective connection guides:
| Connection | Guide |
| ---------- | ------------------------------------------- |
| PostgreSQL | [PostgreSQL](/datasource-config/postgresql) |
| MySQL | [MySQL](/datasource-config/mysql) |
| SQL Server | [SQL Server](/datasource-config/mssql) |
| Oracle | [Oracle](/datasource-config/oracle) |
| Snowflake | [Snowflake](/datasource-config/snowflake) |
## Choosing a format
| Use case | Recommended format |
| ---------------------------------------------------- | ------------------------------------- |
| Human-readable exports from spreadsheets or ETL jobs | CSV or TSV |
| Vendor pipe-delimited daily drops | PSV (2.1+) or CSV with `sep_override` |
| Large historical archives, column pruning | Parquet |
| Low-latency interchange between Arrow-native tools | Arrow IPC / Feather |
| Excel exports from business users | XLSX |
| Scientific simulation output | HDF5 |
| Hierarchical vendor XML feeds | XML |
| SQL warehouse tables already in your estate | Matching database connection |
| Wire-level FIX tick replay | PCAP / PCAPng (2.2+) |
## Related
Connection + dataset architecture and shared `data_args` fields
Full `data_args` reference for file-based sources
# Azure Blob Storage
Source: https://knowledge.cloudquant.com/integrations/azure-blob-storage
Retrieving access keys for Azure Blob Storage connections with CloudQuant Data Liberator
# Azure Blob Storage connections
This guide explains how to retrieve and configure Azure Blob Storage access keys for use with the CloudQuant Data Liberator platform.
## Prerequisites
* An Azure account with a Storage Account
* Access to the Azure Portal
* Your CloudQuant account credentials
## Retrieving access keys
### Step 1: Navigate to your storage account
1. Sign in to the [Azure Portal](https://portal.azure.com)
2. Navigate to **Storage accounts**
3. Select the storage account you want to connect
### Step 2: Find access keys
1. In the storage account menu, select **Security + networking** > **Access keys**
2. You will see two keys: **key1** and **key2**
3. Select **Show** to reveal the key values
### Step 3: Copy required information
You will need:
* **Storage account name** — The name of your Azure Storage Account
* **Access key** — Either key1 or key2
* **Container name** — The blob container holding your data
Keep your access keys secure. Never commit them to source control or share them in plain text.
## Configuring the connection
Provide the following to CloudQuant for CloudQuant Data Liberator connectivity:
```
Storage Account: your_storage_account_name
Access Key: your_access_key
Container: your_container_name
```
## Connection string format
If configuring programmatically, the connection string format is:
```
DefaultEndpointsProtocol=https;AccountName=your_account;AccountKey=your_key;EndpointSuffix=core.windows.net
```
## Key rotation
Azure provides two keys to enable rotation without downtime:
1. Update your applications to use **key2**
2. Regenerate **key1**
3. Update applications to use the new **key1**
4. Regenerate **key2**
Rotate your access keys regularly as a security best practice.
## Troubleshooting
| Issue | Solution |
| --------------------- | ----------------------------------------------------------------- |
| Authentication failed | Verify the access key hasn't been rotated |
| Container not found | Check the container name and ensure it exists |
| Network error | Verify firewall rules allow access from CloudQuant Data Liberator |
# Grafana
Source: https://knowledge.cloudquant.com/integrations/grafana
Connect your Grafana instance to CloudQuant Data Liberator's Prometheus stack using a rotatable Bearer token
# Grafana integration
CloudQuant Data Liberator exposes a curated, read-only slice of its internal Prometheus stack at a Bearer-authenticated endpoint. You can wire your own Grafana (or any Prometheus-API-compatible client) to the same metrics that power the in-product **System Monitoring** view.
This integration is intended for ops teams, on-call rotations, and infrastructure dashboards that need to live alongside metrics from systems outside CloudQuant.
## At a glance
| | |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Endpoint** | `https:///metrics-api-bearer/` |
| **Authentication** | `Authorization: Bearer cqm_<48-hex-chars>` |
| **Protocol** | Standard Prometheus HTTP API (`/api/v1/query`, `/api/v1/query_range`, `/api/v1/series`, `/api/v1/labels`, `/api/v1/label//values`, `/api/v1/status/buildinfo`, `/federate`) |
| **Surface** | Read-only. Admin endpoints (`/api/v1/admin/*`) and remote-write are gated off at the Prometheus server, not just at the gateway. |
| **Token management** | Liberator UI → **System Monitoring → Grafana Integration** tab |
## Prerequisites
* A user with **super-admin** privileges on your Liberator instance (token management is a super-admin operation).
* Network reachability from your Grafana host to your Liberator hostname over HTTPS.
* Grafana 9.x or newer (any version that supports custom HTTP headers on the Prometheus datasource).
## Step 1 — Issue a Bearer token
1. Sign in to the Liberator UI as a super-admin.
2. Open **System Monitoring** from the top navigation.
3. Click the **Grafana Integration** action in the upper-right of the page.
4. Click **Generate token**. The new token is shown **exactly once**, so copy it immediately.
Tokens follow the format `cqm_` + 48 hex characters. Every token issued through this dialog is logged with the issuing user and timestamp; tokens can be listed and revoked from the same dialog.
The full token value is only visible at issue time. Treat it like a password: store it in your secrets manager and never commit it to source control. If a token is lost, revoke it and issue a new one rather than trying to recover it.
Issue one token per logical consumer (e.g. `grafana-prod`, `federated-prometheus-eu`) so you can revoke a single consumer's access without disrupting others.
## Step 2 — Configure the Grafana datasource
### Through the Grafana UI
1. **Connections → Data sources → Add data source → Prometheus.**
2. Set the fields below:
| Field | Value |
| ------------------------- | ------------------------------------------------------- |
| **Name** | `cloudquant-liberator` (or any label you prefer) |
| **Prometheus server URL** | `https:///metrics-api-bearer` |
| **HTTP method** | `POST` (recommended, handles longer queries) |
| **Scrape interval** | `30s` (matches the upstream Prometheus scrape interval) |
3. Under **Custom HTTP Headers**, click **+ Add header** and set:
* **Header** = `Authorization`
* **Value** = `Bearer cqm_`
4. Click **Save & test**. You should see **"Successfully queried the Prometheus API."**
### Through provisioning (recommended for production)
For reproducible setups, drop this file at `/etc/grafana/provisioning/datasources/cloudquant.yaml`:
```yaml theme={null}
apiVersion: 1
datasources:
- name: cloudquant-liberator
uid: cloudquant-liberator
type: prometheus
access: proxy
url: https:///metrics-api-bearer
isDefault: true
editable: true
jsonData:
httpMethod: POST
timeInterval: 30s
prometheusType: Prometheus
httpHeaderName1: Authorization
secureJsonData:
httpHeaderValue1: "Bearer cqm_"
```
Restart Grafana (or send `SIGHUP`) to pick up the file. The token lives in `secureJsonData`, which Grafana persists encrypted at rest.
## Step 3 — Run a query
Once the datasource is healthy, you can browse it in **Explore** or build dashboards against it. A few queries to verify end-to-end connectivity:
```promql theme={null}
# All scrape targets currently up
sum(up)
# Request rate by HTTP status class through the Liberator gateway
sum by (envoy_response_code_class) (rate(envoy_http_downstream_rq_xx[5m]))
# Liberator waiting-room active connections
liberator_waitingroom_connections_active
# p95 upstream request latency
histogram_quantile(
0.95,
sum by (le) (rate(envoy_cluster_external_upstream_rq_time_bucket[5m]))
)
```
## What's exposed
The Bearer endpoint forwards to the same Prometheus server that powers the in-product **System Monitoring** view, so anything you see there is also queryable here. Notable metric families:
| Prefix | What it covers |
| ----------------- | ---------------------------------------------------------------------------------- |
| `envoy_*` | Envoy Gateway — request rates, latency histograms, upstream connection health |
| `liberator_*` | Liberator application — waiting-room queue, in-flight query times, exporter health |
| `cache_manager_*` | Cache-manager worker pool status and error counters |
| `kube_*` | kube-state-metrics — pods, deployments, nodes, conditions |
| `container_*` | cAdvisor — per-container CPU, memory, filesystem, network |
| `node_*` | node-exporter — host CPU, memory, disk, network |
| `prometheus_*` | The Prometheus server itself — useful for sizing dashboards |
For a complete metric inventory in your environment, query `/api/v1/label/__name__/values` once you've connected.
## Rotating a token
1. Open the **Grafana Integration** dialog as in Step 1.
2. Issue a **new** token for the same logical consumer.
3. Update the new token in Grafana (UI: edit the datasource → swap the `Authorization` header value; provisioning: replace `httpHeaderValue1` and restart).
4. Wait until your Grafana datasource health check passes with the new token.
5. **Revoke** the old token from the same dialog.
This zero-downtime pattern is the same as the [AWS access-key rotation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey) workflow.
## Limitations
* **Read-only.** Prometheus admin endpoints (`/api/v1/admin/tsdb/*`) and the remote-write receiver are not exposed. Even tokened callers cannot mutate the TSDB through this endpoint.
* **One Prometheus.** The endpoint serves the Liberator cluster's primary Prometheus only. Federated views across multiple Liberator clusters require Prometheus-side federation on the consumer side; use `/metrics-api-bearer/federate` with appropriate `match[]` parameters.
* **CORS.** Browser-side direct calls are not supported (the gateway only allows server-side calls). Grafana proxies through its backend, so this restriction does not affect dashboard use.
## Troubleshooting
| Symptom | Diagnosis | Resolution |
| -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Save & test` returns *"non-JSON: `...`"* | The request is being redirected to the OIDC login page — the `Authorization` header isn't being sent, or the token is wrong. | Verify the header name is exactly `Authorization` and the value begins with `Bearer ` (with a trailing space) before the token. |
| `Save & test` returns *401 Unauthorized* | Token has been revoked, never issued, or has a typo. | Re-issue from the **Grafana Integration** dialog and update the datasource. |
| `Save & test` returns *403 Forbidden* | The Liberator gateway rejected the request before reaching Prometheus. | Check with your CloudQuant administrator that the bearer route is enabled on your cluster. |
| Health check passes but every query returns *"no data"* | Time range is outside Prometheus's retention window, or the metric name has changed in a recent upgrade. | Open **Explore**, hit `/api/v1/label/__name__/values` against the datasource, and confirm the metric exists. |
| Range queries fail with *"context deadline exceeded"* | Query window × resolution exceeds the gateway's response timeout. | Use the in-product **System Monitoring** view as a sanity check on the query, then widen your Grafana panel's `interval` or narrow the time range. |
# Integrations Overview
Source: https://knowledge.cloudquant.com/integrations/overview
Connect CloudQuant Data Liberator to AI assistants and observability tools
# Integrations
Connect Liberator to external AI assistants and observability tools. AI clients use the Liberator **MCP** endpoint; Grafana uses the Bearer-authenticated metrics API.
## AI assistants (MCP)
Available in Liberator **2.3** and later. Point your client at the Liberator MCP Endpoint URL from the Liberator UI (for example, `https://api.cloudquant.ai/mcp`), then follow the setup guide for your assistant.
MCP connector with OAuth in ChatGPT Apps
MCP tool with OAuth 2.0 Dynamic discovery
Install the Liberator .mcpb extension bundle
## Observability
Available in Liberator **2.2** and later.
Prometheus metrics via a rotatable Bearer token
## Related
Configure Liberator data source connections and datasets
In-product cluster, queue, dataset, and usage views
# S3 Bucket Setup
Source: https://knowledge.cloudquant.com/integrations/s3-bucket-setup
How to set up your S3 bucket so that CloudQuant Data Liberator can connect and read data from it
# S3 bucket setup for CloudQuant Data Liberator
This guide walks you through setting up your Amazon S3 bucket for use with the CloudQuant Data Liberator platform.
## Prerequisites
* An AWS account with S3 access
* IAM permissions to create buckets and manage policies
* Your CloudQuant account credentials
## Step 1: Create an S3 bucket
1. Navigate to the [AWS S3 Console](https://s3.console.aws.amazon.com/)
2. Select **Create bucket**
3. Select a unique bucket name and region
4. Configure bucket settings as needed
## Step 2: Configure bucket policy
Add a bucket policy that grants CloudQuant Data Liberator read access:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::CLOUDQUANT_ACCOUNT_ID:root"
},
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::your-bucket-name",
"arn:aws:s3:::your-bucket-name/*"
]
}
]
}
```
Contact CloudQuant support for the correct AWS Account ID to use in the bucket policy.
## Step 3: Configure CORS (if needed)
If accessing data from web applications, configure CORS:
```json theme={null}
[
{
"AllowedHeaders": ["*"],
"AllowedMethods": ["GET"],
"AllowedOrigins": ["*"],
"ExposeHeaders": []
}
]
```
## Step 4: Generate access keys
1. Navigate to IAM in the AWS Console
2. Create a new IAM user or use an existing one
3. Generate an Access Key ID and Secret Access Key
4. Provide these credentials to CloudQuant securely
## Step 5: Verify connectivity
Contact CloudQuant support to verify that CloudQuant Data Liberator can successfully connect to your S3 bucket.
## Troubleshooting
| Issue | Solution |
| ---------------- | ---------------------------------------------- |
| Access Denied | Verify bucket policy and IAM permissions |
| Bucket Not Found | Confirm bucket name and region |
| Timeout | Check network connectivity and security groups |
# Introduction
Source: https://knowledge.cloudquant.com/introduction
Welcome to the CloudQuant Documentation - your comprehensive guide to the CloudQuant Data Liberator data platform
# Welcome to CloudQuant
CloudQuant provides institutional-grade financial data infrastructure through the **CloudQuant Data Liberator** platform — a simple point-in-time data access API for live or historical time series data.
## What is CloudQuant Data Liberator?
CloudQuant Data Liberator is CloudQuant's data delivery platform that provides:
* **Multi-language API support** — Python, C#, JavaScript, Java, R, RESTful, C++, and Excel
* **Time series data access** — Query historical and live financial datasets
* **70+ connectors** — so you don't have to move your data to get insights
* **Cloud-native integrations** — S3, Azure Blob Storage, and direct connectivity
## Quick start
Get started with CloudQuant Data Liberator for Python — the most popular way to access data
Explore the full API reference for all supported languages
Browse 70+ available datasets and data integrations
Configure connections to databases, cloud storage, and file systems
## Platform components
| Component | Description |
| --------------------------------- | -------------------------------------------------- |
| **CloudQuant Data Liberator API** | Core data access API with multi-language support |
| **Data Catalog** | 70+ integrated financial and alternative datasets |
| **Integrations** | Cloud platform connectivity (S3, Azure) |
| **Datasource Config** | Connect databases, cloud storage, and file systems |
## Need help?
* Visit the [CloudQuant Support Portal](https://www.cloudquant.com/tickets) for direct assistance
* Browse the documentation sections using the sidebar navigation
# Network Configuration
Source: https://knowledge.cloudquant.com/network-configuration
Configure your network environment for CloudQuant Data Liberator access
# Network configuration
This guide covers the network setup required to connect to the CloudQuant Data Liberator platform.
## Firewall requirements
Ensure the following outbound connections are permitted:
| Protocol | Port | Destination | Purpose |
| -------- | ------- | --------------------------------------------- | ------------------------------- |
| HTTPS | 443 | CloudQuant API endpoints | Data queries and authentication |
| TCP | Various | CloudQuant Data Liberator streaming endpoints | Live data feeds |
## Proxy configuration
If your organization uses a proxy server, configure the CloudQuant Data Liberator client accordingly:
```python Python theme={null}
import os
os.environ['HTTPS_PROXY'] = 'http://your-proxy:8080'
```
```bash Shell theme={null}
export HTTPS_PROXY=http://your-proxy:8080
export NO_PROXY=localhost,127.0.0.1
```
## DNS requirements
Ensure your DNS can resolve CloudQuant's service endpoints. Contact CloudQuant support for the current list of required domains.
## Troubleshooting
If you experience connectivity issues:
1. Verify outbound HTTPS (port 443) is open
2. Check proxy settings if applicable
3. Ensure DNS resolution is working
4. Contact [CloudQuant Support](https://www.cloudquant.com/tickets) for endpoint-specific assistance
# CloudQuant Data Liberator Overview
Source: https://knowledge.cloudquant.com/overview
Understanding the CloudQuant Data Liberator data platform architecture and capabilities
# CloudQuant Data Liberator overview
CloudQuant Data Liberator is CloudQuant's core data delivery platform that provides simple point-in-time data access via API for both live and historical time series data.
## Architecture
CloudQuant Data Liberator serves as the central hub for accessing CloudQuant's data ecosystem:
* **Data Ingestion** — Connects to 70+ data providers and integrates their feeds
* **Data Storage** — Manages time series data with efficient compression and indexing
* **Data Delivery** — Provides API access in multiple programming languages
* **Access Control** — Manages dataset entitlements and user permissions
## Supported languages
CloudQuant Data Liberator provides native client libraries for:
* **Python** — Full-featured client with Pandas DataFrame support
* **C#** — .NET client library
* **JavaScript** — Node.js module
* **Java** — Java client library
* **R** — R language client
* **RESTful** — HTTP REST API for any language
* **C++** — High-performance native client with Apache Arrow support
* **Excel** — Plug-in for direct spreadsheet access
## Key concepts
### Datasets
A dataset is a named collection of time series data. Use `liberator.datasets()` (Python) or equivalent methods to discover available datasets.
### Queries
Queries retrieve data from datasets using parameters like time range (`as_of`, `back_to`), symbols, and column selection.
### Timestamps and MUTS
CloudQuant Data Liberator uses microsecond timestamps (MUTS) for precise time series data alignment.
## Getting started
Choose your preferred language from the [API Reference](/api-reference/concepts/queries-large-datasets) section, or follow the [Python Getting Started Guide](/python-guide/getting-started) for the most common path.
# Understanding as_of and back_to
Source: https://knowledge.cloudquant.com/python-guide/as-of-and-back-to
Learn how the as_of and back_to parameters work with point-in-time datasets in CloudQuant Data Liberator.
# Understanding as\_of and back\_to
The `as_of` and `back_to` parameters define the start and end points for data retrieval. Rather than using "start" and "end," these terms accommodate datasets that are **point-in-time** — meaning different data may be returned based on the `as_of` date.
## What are point-in-time datasets?
Point-in-time datasets reflect data as it existed at a specific historical moment. A common example is revised earnings reports: if your `as_of` date precedes the revision date, you receive original values; if it matches or follows the revision, you get updated figures.
Machine learning datasets increasingly use point-in-time logic. Providers may retrain models and recalculate historical values. You typically want access to the original data from your trading period, and you adopt modified data only when you choose to.
Any point-in-time datasets will be labeled clearly as such in the dataset description in `liberator.datasets()`.
## Source of point-in-time data problems
Standard databases prioritize current information, which introduces **survivorship bias** into quantitative analysis. For instance:
* **SEC filings** — Historical earnings may have been restated due to accounting errors, regulatory scrutiny, or fraud. You need the original reported numbers alongside modification timestamps.
* **Index constituents** — The S\&P 500's composition changes over time. Analysis based on today's index membership creates survivorship bias. For rigorous analysis, you need exact historical index compositions and all changes.
Point-in-time data discipline prevents forward-looking bias that compromises quantitative research integrity.
# Usage of the as_of Parameter
Source: https://knowledge.cloudquant.com/python-guide/as-of-parameter
How to use the as_of parameter to retrieve data as it was known at a particular point in time.
# Usage of the as\_of parameter
The `as_of` parameter allows you to query historical data from a specific moment. You can specify it as a date alone (e.g., `'2024-07-08'`) which CloudQuant Data Liberator interprets as midnight that date, or with specific times down to nanosecond precision.
All CloudQuant Data Liberator times are New York Time (US ET).
## Default behavior
When you omit the `as_of` parameter, CloudQuant Data Liberator assumes the current date and time.
In Liberator **2.4** and later, a future `as_of` on a live-appending dataset is rejected or bounded instead of holding a query worker in an empty-fetch loop. Use `as_of` at or before "now" unless you are intentionally querying forward-dated records that already exist in the source.
## Example query
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-10',
back_to = '2024-07-10',
symbols = ['GOOG', 'AAPL', 'MSFT']))
```
This returns data for the specified date range.
## Without back\_to parameter
If you specify `as_of` but omit or set `back_to` to `None`, the query returns data through yesterday's completed trading bar:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-18',
symbols = ['GOOG', 'AAPL', 'MSFT']))
```
## Jupyter notebook configuration
To display more rows in Jupyter:
```python theme={null}
import pandas as pd
pd.set_option('display.max_rows', 400)
```
Consider your data volume and test with smaller queries initially. See [Query Considerations](/python-guide/query-considerations) for more details.
# Usage of the back_to Parameter
Source: https://knowledge.cloudquant.com/python-guide/back-to-parameter
How to use the back_to parameter to define the start of a time range in CloudQuant Data Liberator queries.
# Usage of the back\_to parameter
The `back_to` parameter is a datetime parameter in CloudQuant Data Liberator queries that defines the start point of a time range for data retrieval.
## Parameter format
You can specify `back_to` in two ways:
**Date only:**
```
'2024-07-08'
```
CloudQuant Data Liberator interprets this as midnight: `'2024-07-08 00:00:00.000000'`
**Specific time:**
You can set it to any time down to nanosecond precision.
## Important notes
All CloudQuant Data Liberator times are New York Time (US ET).
Start with small queries to understand data amounts before requesting large date ranges.
* If `back_to` is omitted or set to `None`, CloudQuant Data Liberator returns the most recent value at the specified `as_of` time.
## Default behavior example
Without `back_to`, the query returns the latest available data:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2023-12-25',
symbols = ['GOOG', 'AAPL', 'MSFT']
))
```
Since December 25, 2023 was not a trading day, CloudQuant Data Liberator returned data from the most recent trading day (December 22, 2023).
# Batch Downloading Data
Source: https://knowledge.cloudquant.com/python-guide/batch-downloading
How to efficiently download large datasets from CloudQuant Data Liberator using batch processing and chunked downloads.
# Batch downloading data
Batch downloading is especially useful when you need to download extremely large datasets. This approach offers several advantages over traditional query methods.
## Key advantages
* **Streamed delivery** — Data arrives as a stream rather than waiting for complete server-side processing
* **Resumable downloads** — Write data to file as it arrives, allowing restart points if connection interrupts
* **Memory efficiency** — Avoids double memory usage common with standard DataFrame downloads
* **Simplicity** — Code length is comparable to standard query-to-DataFrame approaches
## Small batch download example
The simplest implementation submits a single large query and writes batches to file:
```python theme={null}
import liberator, time
start_time = time.time()
for batch in liberator.query(name = 'minute_bars', symbols = None,
as_of = '2024-07-01', back_to = '2024-01-01'):
batch.to_pandas().to_csv("minute_bar_data.csv", mode='a')
print("The query + saving took", (time.time() - start_time)/60.0, " minutes to run")
```
An enhanced version prevents empty files and handles headers correctly:
```python theme={null}
import liberator, time
start_time = time.time()
for i, batch in enumerate(liberator.query(name='daily_bars', as_of = '2024-07-01',
back_to = '2024-01-01', symbols = None)):
if not len(batch):
continue
batch.to_pandas().to_csv('daily_bars.csv', mode='a' if i else 'w',
header=False if i else True)
print("The query + saving took", (time.time() - start_time)/60.0, " minutes to run")
```
## Large batch download process
For very large datasets, splitting downloads into monthly chunks prevents resource constraints:
```python theme={null}
# Define the start and end dates and other setup info for the loop
start_year = 2018
start_month = 3
end_year = 2024
end_month = 06
symbols = None
header = True
individual = 0
header_in_individual = 0
dataset = 'minute_bars'
import liberator, time
back_to_year = start_year
back_to_month = start_month
old_month = None
while (back_to_year < end_year) or (back_to_year == end_year and back_to_month <= end_month):
if back_to_month == 12:
as_of_month = 1
as_of_year = back_to_year + 1
else:
as_of_month = back_to_month + 1
as_of_year = back_to_year
back_to = f"{back_to_year:04d}-{back_to_month:02d}-01"
as_of = f"{as_of_year:04d}-{as_of_month:02d}-01"
print(f"Downloading {dataset} {back_to} to {as_of}", end='')
start_time = time.time()
head = header if (old_month==None) else False
for batch in liberator.query(name=dataset, as_of = as_of,
back_to = back_to, symbols = symbols):
if not len(batch):
continue
if individual:
batch.to_pandas().to_csv(dataset+'_'+back_to+'_'+as_of+'.csv',
mode='a', header=header_in_individual or head)
else:
batch.to_pandas().to_csv(dataset+'_all.csv', mode='a', header=head)
print(" Query + save took", (time.time() - start_time)/60.0, " minutes to run. head:",head)
back_to_year = as_of_year
back_to_month = as_of_month
old_month = as_of_month
```
This monthly chunking approach allows downloading even the largest datasets without memory constraints, making it ideal for production data pipelines.
# Selecting Specific Columns
Source: https://knowledge.cloudquant.com/python-guide/column-selection
How to use the fields parameter to restrict which columns are returned in CloudQuant Data Liberator query results.
# Selecting specific columns
CloudQuant Data Liberator automatically returns a minimum default set of columns for each dataset, which varies but typically includes `_seq`, `muts`, `timestamp`, and `symbol`. Since some datasets contain hundreds of columns, you may want to limit results to specific fields, especially on slower connections.
## Filtering columns with the fields parameter
To reduce returned columns, pass the `fields` parameter with your desired column list:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-22',
symbols = ['AAPL', 'GOOGL'],
fields = ['Open', 'Close']
))
```
CloudQuant Data Liberator places your selected columns at the front of the DataFrame, followed by the default columns.
## Discovering available columns
To identify column names in a dataset, use one of these approaches:
### Using the schema function
```python theme={null}
liberator.datasets(schema=True)['nameOfDataset']
```
### Inspecting a sample query result
```python theme={null}
df.columns
list(df.columns)
print(list(df.columns))
```
Example output for `daily_bars`:
```python theme={null}
['_seq', '_dsname', 'timestamp', 'msg_len', 'msg', 'muts', 'symbol',
'length', 'open', 'high', 'low', 'close', 'volume', 'vwap', 'bvwap',
'spread', 'bidvol', 'askvol', 'count', 'avgdelta', 'Date', 'Time',
'Hour', 'DateTime', 'DateHour']
```
# Checking Dataset Access
Source: https://knowledge.cloudquant.com/python-guide/dataset-access
How to discover which datasets you have access to and view their schemas and descriptions.
# Checking dataset access
CloudQuant Data Liberator provides built-in commands to check your dataset permissions, view available datasets, and access detailed schema information.
## Commands
### Browse the catalog
```python theme={null}
liberator.datasets()
```
Returns datasets visible in the catalog for your account — public datasets and those you are entitled to. Private datasets you are not entitled to are hidden. See [Dataset visibility](/administration/dataset-visibility).
### View entitled datasets only
```python theme={null}
liberator.datasets(entitled=True)
```
Returns datasets you are individually entitled to (in addition to public datasets, depending on client defaults).
### Get dataset details
```python theme={null}
liberator.datasets(details=True)('daily_bars')
```
Retrieves a detailed description of a specific dataset. Replace `'daily_bars'` with your target dataset name.
In Liberator **2.4** and later, details also include catalog-card health: date range covered, symbol count, status (for example, Ready), and last updated. Administrators can configure how often symbol discovery refreshes so new keys appear here after the source changes.
Super Admins can also query **`usage_log`**, a private per-query usage dataset that does not appear in the standard catalog. See [Query usage log](/administration/usage-log).
### View dataset schema
```python theme={null}
liberator.datasets(schema=True)('daily_bars')
```
Returns the complete data dictionary and schema for a specific dataset. Replace `'daily_bars'` as needed.
### Fetch concordance metadata (2.3+)
```python theme={null}
liberator.metadata(name='daily_bars', summary=True, symbol=False)
```
Returns concordance-backed row counts, symbol counts, and start/end timestamps without downloading source rows. See [Dataset metadata](/api-reference/concepts/dataset-metadata) for the `summary` × `symbol` grain matrix.
Use these commands to explore available data before writing queries. The schema information is especially useful for identifying column names to use with the [fields parameter](/python-guide/column-selection).
# Getting Started with CloudQuant Data Liberator for Python
Source: https://knowledge.cloudquant.com/python-guide/getting-started
Learn how to set up and configure the CloudQuant Data Liberator Python library to connect to CloudQuant's data platform.
# Getting started with CloudQuant Data Liberator for Python
This guide introduces the CloudQuant Data Liberator library, available across multiple programming languages via the CloudQuant API platform.
## Basic setup
### Quick start
For environments where CloudQuant Data Liberator is pre-installed, simply use:
```python theme={null}
import liberator
```
### Installation steps
If adding CloudQuant Data Liberator to your own Python environment (local installation, Jupyter, or IDE):
1. Download the required files from your Liberator Profile page
2. Install dependencies:
```bash Conda theme={null}
conda create -n liberator python=3.7
conda activate liberator
conda install -c anaconda requests pyopenssl cryptography six pytz
conda install pandas
conda install -c conda-forge httpx async_generator pyarrow=8.0.0
```
```bash pip theme={null}
python -m venv liberator
source liberator/bin/activate # Windows: liberator\Scripts\activate
pip install requests pandas httpx async_generator "pyarrow==8.0.0" pyOpenSSL cryptography six pytz
```
3. Place files in either:
* The same folder as your code, **or**
* A central Python location (`liberator.py` only — see note below)
Putting `liberator.py` in `site-packages` lets you `import liberator` from anywhere, but `liberator.json` is still read from your working directory unless you set `liberator.auth = '/path/to/liberator.json'`.
To find your Python's site-packages directory:
```python theme={null}
import site
print(site.getsitepackages())
```
### Required files
| File | Purpose |
| ---------------- | -------------------------------------------------------------- |
| `liberator.py` | The Python library |
| `liberator.json` | Pre-populated credentials (username and token) |
| `liberator.pfx` | Security certificate (only required for releases prior to 2.0) |
As of release version 2.0, `liberator.pfx` is no longer required. If you are on a release prior to 2.0, you must also include `liberator.pfx`.
## Server configuration
Files downloaded from your Liberator Profile page include the correct server URL preconfigured in the client.
### Override server URL
To connect to a different instance, set the URL explicitly:
```python theme={null}
liberator.url = 'https://mywidgets.cloudquant.ai/'
liberator.url = 'http://10.10.56.127:47753/'
```
Use `https://` for domain names and `http://` for direct IP addresses. Specify port numbers as shown above when required.
Once your client is configured, you are ready to begin pulling data.
# Accessing Data from a Live Trading Environment
Source: https://knowledge.cloudquant.com/python-guide/live-trading-environment
How to use CloudQuant Data Liberator data in live algorithmic trading environments, including data transfer strategies and live streaming.
# Accessing data from a live trading environment
This guide addresses how to use CloudQuant Data Liberator data in live algorithmic trading environments, particularly when third-party APIs cannot be integrated directly.
## Key challenges
Some organizations restrict third-party APIs in live trading systems. When this applies to you, retrieve the data locally and transfer it to the trading server through approved methods.
## Data transfer solutions
### Low frequency approaches
For infrequently updated data, set up shared file storage accessible to both the live trading application and your local servers. Push data to these locations on a schedule, and coordinate with your Operations department to arrange access.
### High frequency solutions
For more frequent intraday updates, messaging services like [ZeroMQ](https://zeromq.org/) are recommended as lightweight alternatives. ZeroMQ operates peer-to-peer without requiring intermediate broker infrastructure.
## Accessing live data from CloudQuant Data Liberator
CloudQuant Data Liberator delivers live datasets with similar simplicity to historical data retrieval. The platform supports seamless transitions between historical and real-time data.
### Example code
```python theme={null}
import liberator
liberator.url = 'https://weather.cloudquant.ai/'
yesterday = '2024-08-01'
res = liberator.query(name = 'us_live_weather', symbols = '78758', as_of = 'live', back_to = yesterday)
for batch in res:
df = batch.to_pandas()
print(df)
```
When `back_to` is specified, the query delivers historical data from that point forward, then automatically transitions to live streaming. Omitting `back_to` initiates live data immediately at the vendor's defined frequency.
# Pulling Data into a Pandas DataFrame
Source: https://knowledge.cloudquant.com/python-guide/pandas-dataframe
How to extract data from CloudQuant Data Liberator datasets into Python pandas DataFrames with practical examples.
# Pulling data into a Pandas DataFrame
This guide demonstrates how to extract data from CloudQuant Data Liberator datasets into Python pandas DataFrames. The examples use `daily_bars`, a dataset containing US Equity daily OHLCV (Open, High, Low, Close, Volume) data with additional fields.
## Example: pulling daily bars for AAPL
### Multi-line approach
```python theme={null}
today = '2024-06-20'
oneMonthAgo = '2024-05-20'
dataset = 'daily_bars'
mySymbols = 'AAPL'
query = liberator.query(name=dataset, symbols=mySymbols, as_of=today, back_to=oneMonthAgo)
df = liberator.get_dataframe(query)
```
### Single-line approach
```python theme={null}
df = liberator.get_dataframe(liberator.query(name='daily_bars', symbols='AAPL', as_of='2024-06-20', back_to='2024-05-20'))
```
## Key features
* **Flexible symbol input** — accepts single symbols, lists, or `None` (all symbols)
* **Granular time selection** — supports daily, minute, second, and nanosecond resolution
* **Simple syntax** — collapsible into single-line queries for efficiency
The query returns a properly formatted pandas DataFrame containing the requested historical data.
# Considerations When Submitting a Query
Source: https://knowledge.cloudquant.com/python-guide/query-considerations
Important constraints, timestamp conventions, and best practices to understand before querying CloudQuant Data Liberator datasets.
# Considerations when submitting a query
Before submitting your first CloudQuant Data Liberator query, there are several important factors to keep in mind regarding constraints, timestamps, and data volume.
## Constraints on a query
The system imposes only two limitations: the **time range** selected and the **symbols** chosen for analysis.
All timestamps in CloudQuant Data Liberator use **New York Time (US ET)**, regardless of the market's geographic location. This standardization helps you understand when data becomes available to you.
## Notes on timestamps
Datasets frequently contain multiple timestamps serving different purposes:
* The **primary timestamp** indicates when you can access the data.
* **Secondary timestamps** may represent exchange or execution times, which typically occur before or coincide with the main timestamp.
For calculated datasets like minute bars, there is often a `bar_time` distinct from the availability timestamp. In US equities markets, the first daily bar concludes at `9:30:59.999999`, with a `bar_time` of `9:31:00.000000`.
Calculated datasets may show vendor timestamps differing from CloudQuant's timestamp. CloudQuant's timestamp is designed to give you the best possible idea of the **actual available time of the data**.
## Think before you query
Before submitting queries, assess the likely data volume.
A simple request for AAPL daily bars for April might return approximately 30 rows. The same parameters applied to NBBO data (all bids and asks) could generate enormous datasets requiring extensive system processing.
Best practice involves querying unfamiliar datasets with **limited date ranges initially**, then progressively expanding timeframes as you become comfortable with the data scale.
In Liberator **2.4** and later, administrators can also set a **maximum number of days** a single query may span on a dataset. If you hit that cap, the error message names the limit and the configured value — narrow `back_to` / `as_of` and retry. See [Fine-grained permissions](/administration/fine-grained-permissions#limits-and-settings).
For guidance, contact [customer\_success@cloudquant.com](mailto:customer_success@cloudquant.com).
# Query Parameter Combinations
Source: https://knowledge.cloudquant.com/python-guide/query-parameters
A summary of all CloudQuant Data Liberator query parameters and how different combinations affect the results you receive.
# Query parameter combinations
This guide explains the main parameters for CloudQuant Data Liberator queries and the expected results from different combinations.
## Core parameters
| Parameter | Description | Default |
| --------- | ------------------------------------------------------------------------------------------ | -------------------- |
| `name` | Dataset identifier (required) | -- |
| `symbols` | Single symbol, list, or (2.4+) `LIKE` pattern with `%`, e.g. `['AAPL','GOOG']` or `'PWR%'` | `None` (all symbols) |
| `back_to` | Query start date | `None` |
| `as_of` | Query end date | Current date/time |
| `fields` | Specify desired columns to reduce download time | All columns |
| `stats` | Set to `'Total'` for symbol counts instead of full results | -- |
| `where` | Custom SQL predicate appended to the generated filters | `None` |
| `sql` | Full raw SQL query (replaces `name`, time range, and other construction params) | `None` |
Use `liberator.datasets(entitled=True)` to list all datasets available to you.
See the [Query Parameters Reference](/api-reference/concepts/query-parameters) for the full parameter list, including superquery, N-Query, and authentication options.
## Query examples
### All parameters included
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-16',
symbols = ['AAPL', 'GOOGL']))
```
Returns data for specified symbols within the date range.
### With a symbol pattern (2.4+)
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'energy_contracts',
as_of = '2026-08-02',
back_to = '2026-08-01',
symbols = 'PWR%'))
```
An unescaped `%` is a `LIKE` wildcard, so this returns every key that starts with `PWR`. Mix exact keys and patterns in one list (`['AAPL', 'PWR%']`). Escape a literal percent as `\%`. See [Pattern filters](/python-guide/symbols-and-key-fields#pattern-filters-24).
### Without symbols
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-16'))
```
Defaults to all available symbols.
### Without `as_of`
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
back_to = '2024-07-22'))
```
Defaults `as_of` to current time, returning data from present back to specified date.
### Without `back_to`
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2023-12-25'))
```
Returns the most recent data point per symbol as of the specified date. Look-back distance varies by dataset density.
### Minimal query (name only)
```python theme={null}
df = liberator.get_dataframe(liberator.query(name = 'daily_bars'))
```
Returns the latest data point for all symbols in the dataset.
## Custom `where` filters
Use `where` with a standard parameter-based query to append extra SQL predicates on top of the filters CloudQuant Data Liberator generates for time range and symbols. The clause is combined with `AND` — do not include the `WHERE` keyword.
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name='daily_bars',
as_of='2024-07-24',
back_to='2024-07-16',
symbols=['AAPL', 'GOOGL'],
where='volume > 1000000'
))
```
`where` works with time-range, LKV, and N-Query modes. Column names that match the dataset schema are automatically quoted when needed.
### Multi-dataset / concordance-style filters
For datasets keyed by more than symbol (for example concordance tables with a `dataset` column), use `where` to narrow the result set:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name='concordance',
as_of='2025-01-01',
back_to='2024-01-01',
where="dataset = 'my_derived_dataset'"
))
```
`where` cannot be combined with `sql`. Use one approach or the other.
## Raw `sql` queries
Pass a complete SQL `SELECT` statement via `sql` instead of `name`, `back_to`, `as_of`, and related parameters. When `sql` is present, parameter-based query construction is skipped.
```python theme={null}
df = liberator.get_dataframe(liberator.query(
sql='''
SELECT *
FROM "daily_bars"
WHERE muts >= 1704067200000000
AND muts < 1704153600000000
AND symbol = $$AAPL$$
''',
user='my_user',
system='API'
))
```
### SQL requirements
* The statement must be a **`SELECT`** (including `UNION` / `INTERSECT` / `EXCEPT` branches).
* Every `SELECT` arm must include an explicit **`WHERE`** clause.
* By default, the `WHERE` clause must include a **time-range predicate** on `muts` or another configured column (`=`, `>`, `>=`, `<`, `<=`, or `BETWEEN`). Use microsecond timestamps for `muts` values.
* Quote dataset names with double quotes when they contain special characters.
* Use `$$symbol$$` dollar-quoting for string literals in filters.
### Parameters allowed with `sql`
When using `sql`, only connection, auth, and transfer options are permitted alongside the statement — for example `user`, `system`, `token`, `compress`, `json_xfer`, and `batch_size`. Supplying non-empty values for `name`, `back_to`, `as_of`, `where`, or other construction parameters raises an error.
Raw SQL bypasses the convenience of `name` / `symbols` / `back_to` / `as_of`. Prefer parameter-based queries unless you need full control over the generated SQL.
# Required Timestamps in Queries
Source: https://knowledge.cloudquant.com/python-guide/required-timestamps
Every Liberator query must include a valid date or time range using back_to and as_of (or an explicit WHERE on time columns)
# Required timestamps in queries
Liberator enforces a **time-based predicate on all queries**. Every query must include a valid, non-zero date or time range. Queries without one return an error and no data.
Previously, a query with no time parameters could return an entire dataset. That behavior was removed to prevent unexpectedly large result sets and unnecessary load.
Review saved notebooks, scripts, and scheduled jobs. Any query that omits `back_to` / `as_of` (or an explicit time `WHERE` in raw `sql`) will fail until updated.
Applies to Liberator **2.1 and later**.
## Valid timestamps
A valid time scope must:
* Cover a **specific date, date range, or datetime range** — not zero, null, or empty.
* Use **`back_to` and `as_of`** for parameter-based queries (Liberator's point-in-time window). See [Understanding as\_of and back\_to](/python-guide/as-of-and-back-to).
* Fall within the dataset's available coverage (valid dates outside coverage return an **empty** result, not a timestamp error).
### Correct examples
```python theme={null}
import liberator
# Single day
df = liberator.get_dataframe(liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2026-01-01",
as_of="2026-01-01",
))
# Date range
df = liberator.get_dataframe(liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2026-01-01",
as_of="2026-06-30",
))
# Datetime range
df = liberator.get_dataframe(liberator.query(
name="my_dataset",
symbols=["AAPL"],
back_to="2026-06-01 09:30:00",
as_of="2026-06-01 16:00:00",
))
```
### Incorrect examples
```python theme={null}
# Missing back_to and as_of — error
liberator.query(name="my_dataset", symbols=["AAPL"])
# Null dates — error
liberator.query(name="my_dataset", symbols=["AAPL"], back_to=None, as_of=None)
```
### Raw `sql` queries
Every `SELECT` must include an explicit **`WHERE`** clause with a time-range predicate on `muts` or another configured time column. See [Query parameters — Raw SQL](/python-guide/query-parameters#raw-sql-queries).
## What the error looks like
The API or client returns an error indicating a time-based predicate is required. This is expected enforcement, not a platform failure.
## Update existing queries
1. **Find affected calls** — Search for `liberator.query()` without `back_to` / `as_of`, or raw `sql` without a time `WHERE`.
2. **Choose the right window** — Daily jobs: parameterize to yesterday or today. Historical pulls: set explicit start and end dates.
3. **Add parameters** — Use the examples above.
4. **Test** — Run manually before re-enabling schedules.
## FAQ
**I used to pull a full dataset for reference. What now?**\
Set `back_to` and `as_of` to the dataset's available range (ask your administrator or check dataset details in the admin portal).
**Error but I provided dates?**\
Confirm values are not null/empty, `back_to` is not after `as_of`, and dates use `YYYY-MM-DD` or `YYYY-MM-DD HH:MM:SS`.
**Valid dates but empty results?**\
The timestamp is accepted; the range may fall outside dataset coverage.
**AI assistant queries?**\
Assistants must specify a time range. Timeless requests should prompt for dates; otherwise the query fails like any other.
## Related
Point-in-time semantics for Liberator datasets
Full parameter reference including raw SQL rules
# Symbols and Key Fields
Source: https://knowledge.cloudquant.com/python-guide/symbols-and-key-fields
Understanding the symbol parameter in CloudQuant Data Liberator queries, how to discover available symbols, and how to query for all symbols in a dataset.
# Symbols and key fields
The **symbol** parameter serves as the key for querying datasets in CloudQuant Data Liberator. The platform initially focused on US Equity Symbols, but its datasets have expanded to include diverse data types such as shipping, weather, and market research. These datasets use varied keys like zip codes and geographic locations. CloudQuant Data Liberator keeps the `symbols` parameter name for consistency with existing integrations.
**Default Value:** `None` (returns all symbols/keys, including blanks/NaN values)
`symbols` and `keys` are synonyms — use either name.
When setting your query parameters, consider the amount of data you may be requesting and start small until you understand data volume patterns for each symbol/date combination.
## Determining unique symbols in a dataset
The most straightforward approach is to query and count the data. Here is an example using the `daily_bars` dataset:
```python theme={null}
df = liberator.get_dataframe(liberator.query(
name = 'daily_bars',
as_of = '2024-07-24',
back_to = '2024-07-22',
symbols = None))
df
```
This returns over 20,000 rows. To count unique symbols:
```python theme={null}
len(df.symbol.unique())
# Output: 10524
```
## Storing and displaying symbols
To store symbols in a variable:
```python theme={null}
allsym = list(df.symbol.unique())
```
To display the complete list (with caution — lists can be lengthy):
```python theme={null}
print(list(df.symbol.unique()))
```
## Query methods for all symbols
Either explicitly set `symbols = None` or omit the parameter entirely, as it defaults to `None`.
## Pattern filters (2.4+)
In Liberator **2.4** and later, `symbols` / `keys` accept SQL `LIKE` patterns. An unescaped `%` matches any sequence of characters, so you can request a product family or naming pattern without listing every contract.
```python theme={null}
# All keys that start with PWR
df = liberator.get_dataframe(liberator.query(
name='energy_contracts',
symbols='PWR%',
back_to='2026-08-01',
as_of='2026-08-02',
))
# Mix exact keys and patterns
df = liberator.get_dataframe(liberator.query(
name='energy_contracts',
symbols=['AAPL', 'MSFT', 'PWR%'],
back_to='2026-08-01',
as_of='2026-08-02',
))
```
| Pattern | Matches |
| ------- | --------------------------------------------- |
| `PWR%` | `PWR`, `PWR-H`, `PWR_CAL_26` |
| `G%GL` | `GOGL`, `GAGL`, `GOOGLE` |
| `A\%PL` | only the literal key `A%PL` (escaped percent) |
Exact keys (no unescaped `%`) still use equality, including lists of tickers. Liberator only switches to `LIKE` for values that contain `%`.
Start with a short date range when you first use a pattern. A leading `%` on a large dataset can return far more rows than an exact-symbol query.
See the [query parameters reference](/api-reference/concepts/query-parameters#dataset--symbol-parameters) for `symbols` and `keys`.
# Security & Access
Source: https://knowledge.cloudquant.com/security-and-access
CloudQuant Data Liberator security model, authentication, and access control
# CloudQuant Data Liberator security and access
CloudQuant Data Liberator provides enterprise-grade security for data access and distribution.
## Authentication
Access to CloudQuant Data Liberator requires valid credentials provided by CloudQuant. Authentication is handled through:
* **API Keys** — Used for programmatic access via client libraries
* **User Credentials** — Used for interactive access via Excel and CQAI
## Dataset entitlements
Access to specific datasets is controlled through entitlements:
* Each user or organization is granted access to specific datasets
* Use `liberator.datasets()` (Python) or equivalent to see your available datasets
* Organization administrators can manage users and entitlements from the admin portal — see [Administration Overview](/administration/overview)
* Contact CloudQuant support to request access to additional datasets
### For administrators
Add, create, deactivate users and grant or revoke dataset access
Restrict query date ranges, row limits, keys, and columns
## Network security
CloudQuant Data Liberator supports secure connectivity through:
* **TLS/SSL encryption** for all data in transit
* **IP whitelisting** for restricted access environments
* **VPN connectivity** for on-premises deployments
## Anti-virus considerations
When running CloudQuant Data Liberator client libraries locally, ensure your anti-virus software does not block:
* CloudQuant Data Liberator client executables
* Network connections to CloudQuant data endpoints
* Local cache directories used by the client
See the [Network Configuration](/network-configuration) page for detailed setup instructions.
# SQL Server (MSSQL)
Source: https://knowledge.cloudquant.com/datasource-config/mssql
Configure Microsoft SQL Server datasources with ODBC support
# SQL Server (MSSQL)
SQL Server datasources use the ODBC driver and Microsoft ODBC Driver 18 for SQL Server.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on SQL Server connections.
## Connection configuration
### URI format
```
mssql://user:password@host:1433/database
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | ------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"mssql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | SQL Server connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (SQL Server default `1433`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Optional fields
| Field | Type | Default | Description |
| ------------ | ------ | ------- | ------------------------------------------------------- |
| `encryption` | string | | TLS encryption mode (e.g., `"yes"`, `"no"`, `"strict"`) |
### Environment variables
| Variable | Description |
| ------------------- | ----------------------------------------------------------------------------------------------- |
| `ODBC_MSSQL_DRIVER` | Override the ODBC driver name (default: `"ODBC Driver 18 for SQL Server"`) |
| `GODEBUG` | Set to `x509negativeserial=1` for servers using older certificates with negative serial numbers |
The `ODBC_MSSQL_DRIVER` environment variable allows you to use a different version of the Microsoft ODBC driver (e.g., Driver 17) without modifying the connection configuration.
### Example connection
```json theme={null}
{
"name": "mssql-market-data",
"connection_type": "mssql",
"connection_sub_type": "database",
"uri": "mssql://libuser:s3cureP@ss@db.example.com:1433/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss",
"encryption": "yes"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "mssql-trades",
"connection_type": "mssql",
"connection_sub_type": "database",
"uri": "mssql://libuser:s3cureP@ss@sqlserver.example.com:1433/trades_db",
"server": "sqlserver.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss",
"encryption": "yes"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "mssql-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "mssql-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native SQL Server `datetime2` or `datetimeoffset` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## Encryption configuration
SQL Server connections support several TLS encryption modes via the `encryption` field:
| Value | Description |
| ---------- | --------------------------------------------- |
| `"yes"` | Require encrypted connection (recommended) |
| `"no"` | Do not encrypt the connection |
| `"strict"` | Strict TLS 1.2+ encryption (SQL Server 2022+) |
When using `encryption: "yes"` with self-signed certificates, you may need to add `TrustServerCertificate=yes` to the connection. For production, always use properly signed certificates.
## Legacy certificate support
For SQL Server instances using older certificates with negative serial numbers:
```bash theme={null}
export GODEBUG=x509negativeserial=1
```
This is common with older SQL Server installations that have not rotated their TLS certificates.
## Driver and dependencies
| Component | Details |
| ---------------- | ------------------------------------------------ |
| Driver | `ODBC driver` with ODBC Driver 18 for SQL Server |
| Protocol | ODBC / TDS |
| Arrow conversion | High-performance ODBC driver |
| ODBC required | Yes (Microsoft ODBC Driver 18) |
| Default port | 1433 |
The Microsoft ODBC Driver 18 must be installed on the CloudQuant Data Liberator host. To use a different driver version, set the `ODBC_MSSQL_DRIVER` environment variable.
## Troubleshooting
### ODBC driver not found
Verify the Microsoft ODBC Driver 18 is installed and registered:
```bash theme={null}
# List installed ODBC drivers
odbcinst -q -d
# Expected output should include:
# [ODBC Driver 18 for SQL Server]
```
Install the driver if missing:
```bash Debian/Ubuntu theme={null}
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/22.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
apt-get update && ACCEPT_EULA=Y apt-get install -y msodbcsql18
```
```bash RHEL/CentOS theme={null}
curl https://packages.microsoft.com/config/rhel/8/prod.repo > /etc/yum.repos.d/mssql-release.repo
ACCEPT_EULA=Y yum install -y msodbcsql18
```
### Using a different driver version
Override the default driver with an environment variable:
```bash theme={null}
export ODBC_MSSQL_DRIVER="ODBC Driver 17 for SQL Server"
```
### Certificate errors (x509)
If you see `x509: certificate signed by unknown authority` or negative serial number errors:
```bash theme={null}
export GODEBUG=x509negativeserial=1
```
### Connection timeout
SQL Server may be configured to listen on a non-default port or require a named instance. Verify connectivity:
```bash theme={null}
# Test port connectivity
nc -zv sqlserver.example.com 1433
# For named instances, use the SQL Browser port (1434/UDP) or specify the port directly
```
### Windows authentication
CloudQuant Data Liberator uses SQL Server authentication (username/password). Windows/Kerberos authentication is not supported through this connection type.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed.
* For partitioned tables, use `dt_regex` to scope queries to relevant partitions.
* Consider using `datetime2` over `datetime` for better precision and performance.
# MySQL
Source: https://knowledge.cloudquant.com/datasource-config/mysql
Configure MySQL datasources with ODBC support
# MySQL
MySQL datasources use the ODBC driver and the MariaDB ODBC connector. The connection string is built internally from the provided fields.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on MySQL connections.
## Connection configuration
### URI format
```
mysql://user:password@host:3306/database
```
The URI is used as a reference identifier. CloudQuant Data Liberator builds the actual ODBC connection string internally from the individual connection fields (`server`, `database`, `username`, `password`).
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | -------------------------------------------------------------------- |
| `connection_type` | string | Must be `"mysql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | MySQL connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (MySQL default `3306`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Optional fields
| Field | Type | Default | Description |
| -------------- | ------ | --------- | ------------------------------------------------------------ |
| `ssl_disabled` | string | `"False"` | Set to `"True"` to disable SSL. Values: `"True"` / `"False"` |
### Example connection
```json theme={null}
{
"name": "mysql-market-data",
"connection_type": "mysql",
"connection_sub_type": "database",
"uri": "mysql://libuser:s3cureP@ss@db.example.com:3306/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss",
"ssl_disabled": "False"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "mysql-trades",
"connection_type": "mysql",
"connection_sub_type": "database",
"uri": "mysql://libuser:s3cureP@ss@db.example.com:3306/trades_db",
"server": "db.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss",
"ssl_disabled": "False"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "mysql-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "mysql-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native MySQL `DATETIME` or `TIMESTAMP` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## SSL configuration
By default, SSL is enabled. To disable SSL (e.g., for local development):
```json theme={null}
{
"ssl_disabled": "True"
}
```
Disabling SSL in production is not recommended. Always use SSL when connecting over untrusted networks. The MariaDB ODBC driver supports TLS 1.2+ by default.
## Driver and dependencies
| Component | Details |
| ---------------- | ------------------------------------ |
| Driver | `ODBC driver` with MariaDB ODBC 3.2+ |
| Protocol | ODBC |
| Arrow conversion | High-performance ODBC driver |
| ODBC required | Yes (MariaDB ODBC connector) |
| Default port | 3306 |
The MariaDB ODBC 3.2+ driver must be installed on the CloudQuant Data Liberator host. This driver is compatible with both MySQL and MariaDB servers. Verify installation with `odbcinst -q -d`.
## Troubleshooting
### ODBC driver not found
If you see errors about missing ODBC drivers, verify the MariaDB ODBC driver is installed and registered:
```bash theme={null}
# List installed ODBC drivers
odbcinst -q -d
# Expected output should include:
# [MariaDB ODBC 3.2 Driver]
```
Install the driver if missing:
```bash Debian/Ubuntu theme={null}
apt-get install libmariadb3 odbc-mariadb
```
```bash RHEL/CentOS theme={null}
yum install MariaDB-connector-odbc
```
### Connection refused
Verify the MySQL server is accepting connections on the specified host and port. Check that the `bind-address` in the MySQL configuration allows remote connections.
```bash theme={null}
mysql -h db.example.com -P 3306 -u libuser -p marketdata
```
### Authentication failed
Ensure the user has been granted access from the CloudQuant Data Liberator host's IP address:
```sql theme={null}
GRANT SELECT ON trades_db.* TO 'libuser'@'liberator-host' IDENTIFIED BY 's3cureP@ss';
FLUSH PRIVILEGES;
```
### SSL handshake errors
If SSL connections fail, check that the MySQL server's SSL certificate is valid and trusted. Set `ssl_disabled` to `"True"` temporarily to confirm the issue is SSL-related.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in MySQL.
* For partitioned tables, ensure `dt_regex` is correctly scoping queries to relevant partitions.
# Oracle
Source: https://knowledge.cloudquant.com/datasource-config/oracle
Configure Oracle Database datasources with Oracle database driver thin mode
# Oracle
Oracle datasources use the `Oracle database driver` in thin mode. No Oracle Instant Client installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on Oracle connections.
## Connection configuration
### URI format
```
oracle://user:password@host:1521/service_name
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | --------------------------------------------------------------------- |
| `connection_type` | string | Must be `"oracle"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | Oracle connection string |
| `server` | string | Database host address |
| `database` | string | Oracle service name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (Oracle default `1521`) |
In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`). Missing tables return a clear error instead of an empty sample.
### Optional fields
| Field | Type | Default | Description |
| ----------- | ------ | ------- | --------------------------------------------- |
| `auth_mode` | string | | Oracle authentication mode (e.g., `"SYSDBA"`) |
| `schema` | string | | Oracle schema name to query against |
### Environment variables
| Variable | Description |
| ----------------------- | ----------------------------------------------- |
| `ORACLE_DEBUG_BOUNDARY` | Enable debug timing for Oracle query boundaries |
### Example connection
```json theme={null}
{
"name": "oracle-market-data",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://libuser:s3cureP@ss@db.example.com:1521/ORCL",
"server": "db.example.com",
"database": "ORCL",
"username": "libuser",
"password": "s3cureP@ss"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database/service name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "oracle-trades",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://libuser:s3cureP@ss@oradb.example.com:1521/TRADEDB",
"server": "oradb.example.com",
"database": "TRADEDB",
"username": "libuser",
"password": "s3cureP@ss",
"schema": "MARKET_DATA"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "oracle-trades",
"data_args": {
"table_options": [
{
"db": "TRADEDB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "oracle-trades",
"batch_size": 10000,
"data_dt_column": "TRADE_TIME",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "SYMBOL"
},
"schema": [
{ "name": "SYMBOL", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "TRADE_TIME", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "PRICE", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "VOLUME", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "EXCHANGE", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Oracle column names are uppercase by default. Use uppercase names in your schema definition and `data_args` fields to match Oracle's metadata.
## Authentication modes
The `auth_mode` field supports Oracle authentication modes for privileged connections:
```json theme={null}
{
"name": "oracle-admin",
"connection_type": "oracle",
"connection_sub_type": "database",
"uri": "oracle://sys:adminP@ss@db.example.com:1521/ORCL",
"server": "db.example.com",
"database": "ORCL",
"username": "sys",
"password": "adminP@ss",
"auth_mode": "SYSDBA"
}
```
Using `SYSDBA` authentication in production datasource connections is not recommended. Create a dedicated read-only user with `SELECT` privileges on the required tables.
## Schema-qualified tables
When the `schema` field is set on the connection, CloudQuant Data Liberator queries tables within that schema. This is useful when the database user's default schema differs from where the data resides:
```json theme={null}
{
"schema": "MARKET_DATA",
"table_options": [
{
"db": "TRADEDB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
]
}
```
This queries `MARKET_DATA.DAILY_TRADES` rather than relying on the user's default schema.
## Microsecond timestamp precision
Oracle `TIMESTAMP(6)` columns support microsecond precision. A custom-patched `oracledb 3.4.0` is available for CloudQuant Data Liberator that preserves full microsecond precision during data extraction.
The standard `Oracle database driver 3.3+` driver supports microsecond timestamps. The custom-patched 3.4.0 build provides additional optimizations for high-frequency timestamp extraction. Contact your CloudQuant Data Liberator administrator for the patched driver.
## Driver and dependencies
| Component | Details |
| ---------------------- | ----------------------------------------- |
| Driver | `Oracle database driver` 3.3+ (thin mode) |
| Protocol | Oracle Net (TNS) |
| Arrow conversion | Via Oracle driver native conversion |
| Oracle Client required | No (thin mode) |
| Default port | 1521 |
Thin mode connects directly to Oracle without requiring Oracle Instant Client. This simplifies deployment and eliminates native library dependencies.
## Troubleshooting
### Connection refused
Verify the Oracle listener is running and accepting connections on the specified host and port:
```bash theme={null}
# Test port connectivity
nc -zv oradb.example.com 1521
# Test with sqlplus (if available)
sqlplus libuser/s3cureP@ss@oradb.example.com:1521/TRADEDB
```
### ORA-12514: TNS listener does not currently know of service
The service name in the URI does not match any registered service on the Oracle listener. Verify the service name:
```sql theme={null}
-- On the Oracle server
SELECT name FROM v$services;
```
### ORA-01017: invalid username/password
Ensure the username and password are correct. Oracle passwords are case-sensitive (depending on the `SEC_CASE_SENSITIVE_LOGON` parameter).
### Table not found (ORA-00942)
Check that the user has `SELECT` privileges on the target table and that the schema is correctly specified:
```sql theme={null}
GRANT SELECT ON MARKET_DATA.DAILY_TRADES TO libuser;
```
### Debug timing
To enable debug timing for Oracle query boundaries, set the environment variable:
```bash theme={null}
export ORACLE_DEBUG_BOUNDARY=1
```
This logs timing information for each query phase, useful for diagnosing performance bottlenecks.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in Oracle.
* For partitioned tables, verify that Oracle partition pruning is being used by checking the execution plan.
* Use `TIMESTAMP(6)` columns for microsecond precision rather than `DATE` (which only supports second precision).
# PostgreSQL
Source: https://knowledge.cloudquant.com/datasource-config/postgresql
Configure PostgreSQL datasources with high-performance native driver support
# PostgreSQL
PostgreSQL datasources use a high-performance native driver for optimized data transfer. No ODBC installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on PostgreSQL connections.
## Connection configuration
### URI format
```
postgresql://user:password@host:5432/database
```
### Required fields
| Field | Type | Description |
| --------------------- | ------------- | ------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"postgresql"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | PostgreSQL connection string |
| `server` | string | Database host address |
| `database` | string | Database name |
| `username` | string | Database user |
| `password` | string | Database password |
| `port` | string or int | Database port. Required in Liberator **2.4+** (PostgreSQL default `5432`) |
The native driver communicates with the database using the PostgreSQL binary protocol and produces data batches directly, avoiding row-by-row conversion overhead. In Liberator **2.4** and later, **Port** is required on Test Connection, and passwords may include URI-special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`).
### Example connection
```json theme={null}
{
"name": "postgres-market-data",
"connection_type": "postgresql",
"connection_sub_type": "database",
"uri": "postgresql://libuser:s3cureP@ss@db.example.com:5432/marketdata",
"server": "db.example.com",
"database": "marketdata",
"username": "libuser",
"password": "s3cureP@ss"
}
```
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "postgres-trades",
"connection_type": "postgresql",
"connection_sub_type": "database",
"uri": "postgresql://libuser:s3cureP@ss@db.example.com:5432/trades_db",
"server": "db.example.com",
"database": "trades_db",
"username": "libuser",
"password": "s3cureP@ss"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "postgres-trades",
"data_args": {
"table_options": [
{
"db": "trades_db",
"table_like": "daily_trades",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "postgres-trades",
"batch_size": 10000,
"data_dt_column": "trade_time",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "symbol"
},
"schema": [
{ "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "trade_time", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "price", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "volume", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "exchange", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Set `data_dt_format` to `true` when using native PostgreSQL `timestamp` or `timestamptz` columns. CloudQuant Data Liberator will auto-detect the datetime type and convert directly without string parsing.
## Partitioned table example
For tables partitioned by date (e.g., `trades_2024_01`, `trades_2024_02`):
```json theme={null}
{
"table_options": [
{
"db": "trades_db",
"table_like": "trades_%",
"token": "trades",
"zone": "America/New_York",
"dt_regex": "trades_(\\d{4})_(\\d{2})",
"dt_format": "%Y_%m"
}
]
}
```
## Driver and dependencies
| Component | Details |
| --------------- | -------------------------- |
| Driver | PostgreSQL native driver |
| Protocol | PostgreSQL binary protocol |
| Data conversion | Native (high-performance) |
| ODBC required | No |
| Default port | 5432 |
The native driver is bundled with CloudQuant Data Liberator. No additional driver installation is needed on the host system.
## Troubleshooting
### Connection refused
Verify the PostgreSQL server is accepting connections on the specified host and port. Check `pg_hba.conf` for client authentication rules.
```bash theme={null}
psql -h db.example.com -p 5432 -U libuser -d marketdata
```
### Authentication failed
Ensure the `username` and `password` in the connection match a valid PostgreSQL role. Check that the role has `CONNECT` privilege on the database and `SELECT` privilege on the target tables.
### SSL/TLS errors
If the server requires SSL, append `?sslmode=require` to the URI:
```
postgresql://libuser:s3cureP@ss@db.example.com:5432/marketdata?sslmode=require
```
Using `sslmode=disable` in production is not recommended. Always use `sslmode=require` or `sslmode=verify-full` when connecting over untrusted networks.
### Empty results from partitioned tables
Verify that `dt_regex` correctly matches the table name suffix and that `dt_format` corresponds to the captured groups. Test the regex against your actual table names.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure the datetime and key columns are indexed in PostgreSQL.
* Use `table_like` to target specific tables rather than scanning all partitions.
# Snowflake
Source: https://knowledge.cloudquant.com/datasource-config/snowflake
Configure Snowflake datasources with native high-performance driver
# Snowflake
Snowflake datasources use the Snowflake native driver for high-performance native data transfer. No ODBC installation is required.
See [Supported Data Formats](/datasource-config/supported-formats) for database source categories and schema column types supported on Snowflake connections.
## Connection configuration
### URI format
```
snowflake://user:password@account/database/schema?warehouse=compute_wh&role=accountadmin
```
### Required fields
| Field | Type | Description |
| --------------------- | ------ | --------------------------------------------------------------------------------------------------------------- |
| `connection_type` | string | Must be `"snowflake"` |
| `connection_sub_type` | string | Must be `"database"` |
| `uri` | string | Snowflake connection string |
| `server` | string | Snowflake account URL (e.g., `"xy12345.us-east-1.snowflakecomputing.com"`) |
| `database` | string | Snowflake database name |
| `username` | string | Snowflake user |
| `password` | string | Snowflake password. In Liberator **2.4+**, special characters (`%`, `?`, `#`, `@`, `/`, `:`, `&`) are supported |
| `account` | string | Snowflake account identifier (e.g., `"xy12345.us-east-1"`) |
| `warehouse` | string | Compute warehouse name |
| `role` | string | Snowflake role |
### Optional fields
| Field | Type | Default | Description |
| -------- | ------ | ------- | --------------------- |
| `schema` | string | | Snowflake schema name |
### Environment variables
| Variable | Description |
| --------------------- | ------------------------------------------------------ |
| `SNOWFLAKE_DEBUGGING` | Enable verbose debug logging for Snowflake connections |
### Example connection
```json theme={null}
{
"name": "snowflake-market-data",
"connection_type": "snowflake",
"connection_sub_type": "database",
"uri": "snowflake://libuser:s3cureP@ss@xy12345.us-east-1/MARKETDATA/PUBLIC?warehouse=COMPUTE_WH&role=DATA_READER",
"server": "xy12345.us-east-1.snowflakecomputing.com",
"database": "MARKETDATA",
"username": "libuser",
"password": "s3cureP@ss",
"account": "xy12345.us-east-1",
"warehouse": "COMPUTE_WH",
"role": "DATA_READER",
"schema": "PUBLIC"
}
```
The Snowflake native driver uses Snowflake's native result set format. Data is returned as batches directly from the Snowflake service with no row-by-row conversion.
## Dataset configuration (data\_args)
### Required fields
| Field | Type | Description |
| ------------------ | ---------------------- | ---------------------------------------------------------- |
| `table_options` | list | Tables to query (see below) |
| `database` | string | Connection reference name |
| `data_dt_column` | string or list | Column(s) containing the datetime value |
| `data_dt_format` | bool or string or list | `true` for native datetime auto-detect, or strptime format |
| `data_dt_timezone` | string | Timezone of the source data |
| `data_key_column` | string or list | Column(s) used as the symbol/key for query filtering |
### Optional fields
| Field | Type | Default | Description |
| --------------- | ---- | ------- | ---------------------------------------- |
| `batch_size` | int | `10000` | Number of rows per query batch |
| `data_dt_nudge` | int | `0` | Microsecond offset applied to timestamps |
### table\_options entry
Each entry in `table_options` describes a table to query:
| Field | Type | Description |
| -------------- | ------ | --------------------------------------------------- |
| `db` | string | Database name |
| `table_like` | string | Table name or pattern |
| `token` | string | Table identifier token |
| `zone` | string | Timezone, default `"America/New_York"` |
| `dt_mode` | string | *(Optional)* Datetime mode |
| `dt_regex` | string | *(Optional)* Regex for partitioned tables |
| `dt_format` | string | *(Optional)* Datetime format for partitioned tables |
| `nudge_micros` | int | *(Optional)* Microsecond nudge |
## Complete example
### Connection
```json theme={null}
{
"name": "snowflake-trades",
"connection_type": "snowflake",
"connection_sub_type": "database",
"uri": "snowflake://libuser:s3cureP@ss@xy12345.us-east-1/TRADES_DB/MARKET?warehouse=COMPUTE_WH&role=accountadmin",
"server": "xy12345.us-east-1.snowflakecomputing.com",
"database": "TRADES_DB",
"username": "libuser",
"password": "s3cureP@ss",
"account": "xy12345.us-east-1",
"warehouse": "COMPUTE_WH",
"role": "accountadmin",
"schema": "MARKET"
}
```
### Dataset
```json theme={null}
{
"name": "us-equity-trades",
"connection": "snowflake-trades",
"data_args": {
"table_options": [
{
"db": "TRADES_DB",
"table_like": "DAILY_TRADES",
"token": "daily_trades",
"zone": "America/New_York"
}
],
"database": "snowflake-trades",
"batch_size": 10000,
"data_dt_column": "TRADE_TIME",
"data_dt_format": true,
"data_dt_timezone": "America/New_York",
"data_dt_nudge": 0,
"data_key_column": "SYMBOL"
},
"schema": [
{ "name": "SYMBOL", "type": "string", "group": "key", "description": "Ticker symbol" },
{ "name": "TRADE_TIME", "type": "string", "group": "time", "description": "Trade timestamp" },
{ "name": "PRICE", "type": "double", "group": "value", "description": "Trade price" },
{ "name": "VOLUME", "type": "int64", "group": "value", "description": "Trade volume" },
{ "name": "EXCHANGE", "type": "string", "group": "value", "description": "Exchange code" }
]
}
```
Snowflake identifiers are uppercase by default. Use uppercase names in your schema definition and `data_args` fields to match Snowflake's metadata, unless the objects were created with double-quoted lowercase names.
## Warehouse and role configuration
The `warehouse` and `role` fields control compute resources and access permissions:
### Warehouse
The warehouse determines the compute cluster used to execute queries. Choose a warehouse sized appropriately for your query workload:
```json theme={null}
{
"warehouse": "COMPUTE_WH"
}
```
Ensure the specified warehouse is set to auto-resume, or queries will fail when the warehouse is suspended. CloudQuant Data Liberator does not issue `ALTER WAREHOUSE ... RESUME` commands.
### Role
The role determines which database objects are accessible:
```json theme={null}
{
"role": "DATA_READER"
}
```
Create a dedicated read-only role for CloudQuant Data Liberator connections rather than using `accountadmin`. Grant `SELECT` on the required databases and schemas to this role.
## Debug logging
Enable verbose debug logging for Snowflake connections to diagnose connectivity or query issues:
```bash theme={null}
export SNOWFLAKE_DEBUGGING=1
```
This produces detailed logs including connection negotiation, query execution timing, and data batch transfer metrics.
## Driver and dependencies
| Component | Details |
| --------------- | --------------------------------------- |
| Driver | `Snowflake native driver` |
| Protocol | Snowflake REST API / streaming protocol |
| Data conversion | Native (high-performance) |
| ODBC required | No |
| Default port | 443 (HTTPS) |
The Snowflake native driver is bundled with CloudQuant Data Liberator. No additional driver installation is needed on the host system. All communication uses HTTPS on port 443.
## Troubleshooting
### Account identifier format
The `account` field must include the region if your account is not in the default AWS us-west-2 region:
```
# US East (Ohio)
xy12345.us-east-1
# EU (Frankfurt)
xy12345.eu-central-1
# Azure (East US 2)
xy12345.east-us-2.azure
```
### Authentication failed (390100)
Verify the username, password, and account identifier. Snowflake passwords are case-sensitive. If multi-factor authentication (MFA) is enabled for the user, you may need to use a service account without MFA.
### Warehouse is suspended
If queries fail with a warehouse suspension error, ensure the warehouse is configured with `AUTO_RESUME = TRUE`:
```sql theme={null}
ALTER WAREHOUSE COMPUTE_WH SET AUTO_RESUME = TRUE;
```
### Role does not have access
Verify the role has the required privileges:
```sql theme={null}
GRANT USAGE ON DATABASE TRADES_DB TO ROLE DATA_READER;
GRANT USAGE ON SCHEMA TRADES_DB.MARKET TO ROLE DATA_READER;
GRANT SELECT ON ALL TABLES IN SCHEMA TRADES_DB.MARKET TO ROLE DATA_READER;
```
### Network connectivity
Snowflake connections require HTTPS (port 443) access to `*.snowflakecomputing.com`. Ensure firewalls and proxy servers allow this traffic.
### Slow queries
* Increase `batch_size` for large result sets (up to `100000`).
* Ensure Snowflake clustering keys align with your datetime and key columns.
* Use an appropriately sized warehouse (e.g., `MEDIUM` or `LARGE`) for large datasets.
* Check that the warehouse is not queued behind other workloads using Snowflake's query history.