SFTP

SFTP (SSH File Transfer Protocol) datasources allow CloudQuant Data Liberator to read CSV, TSV, and other delimited files from remote servers over an encrypted SSH connection. CloudQuant Data Liberator mounts the remote directory via SSHFS/FUSE.

See Supported Data Formats for every file extension Liberator can ingest over SFTP, including formats added in 2.1 and 2.2.

Connection configuration

Required fields

Field	Type	Description
`connection_type`	string	Must be `"sftp"`
`host`	string	SFTP server hostname or IP address
`user`	string	Username for authentication

You must provide either password or key for authentication. If both are specified, key-based authentication takes precedence.

Optional fields

Field	Type	Default	Description
`port`	int	`22`	SSH port number
`password`	string		Password for password-based authentication
`key`	string		SSH private key content (PEM format) for key-based authentication
`prefix`	string	`""`	Remote directory path to use as root
`mount_point`	string		Local mount path for FUSE-based access
`config_name`	string		Internal configuration identifier

Example connection (password authentication)

{
  "name": "sftp-vendor-data",
  "connection_type": "sftp",
  "host": "sftp.vendor.example.com",
  "port": 22,
  "user": "datauser",
  "password": "s3cur3P@ssw0rd",
  "prefix": "/data/daily-feeds/"
}

Example connection (key authentication)

{
  "name": "sftp-internal-data",
  "connection_type": "sftp",
  "host": "data-server.internal.net",
  "port": 2222,
  "user": "liberator-svc",
  "key": "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNza...\n-----END OPENSSH PRIVATE KEY-----",
  "prefix": "/exports/market-data/"
}

Avoid embedding private keys or passwords directly in configuration files. Use environment variables or a secrets manager to inject credentials at deployment time.

Dataset configuration (data_args)

The data_args fields are identical to all file-based sources. See Local File for the full reference. The file_pattern is evaluated relative to the prefix configured on the connection.

Required data_args

Field	Type	Description
`file_pattern`	string	Glob pattern relative to the prefix, e.g., `"*.csv"`
`data_dt_column`	string or list	Column(s) containing the datetime value
`data_dt_format`	string or list	strptime format or special values (`"muts"`, `"uts"`, `"datetime"`)
`data_key_column`	string or list	Symbol/key column(s)

Optional data_args

Field	Type	Default	Description
`sep_override`	string	`","`	Delimiter character
`encoding`	string	`"utf-8"`	File encoding
`data_dt_timezone`	string	`"UTC"`	Source data timezone
`fname_dt_regex`	string		Regex to extract date from filename
`fname_dt_format`	string		strptime format for filename date
`fname_dt_timezone`	string		Timezone of filename date
`fname_dt_nudge`	int	`0`	Microsecond offset for filename date
`fname_dt_approx_seconds`	int		Approximate seconds per file
`arrow_sort`	list	`["symbol", "muts"]`	Sort order
`arrow_timestamp`	bool	`true`	Generate human-readable timestamp column

Complete example

Connection

{
  "name": "sftp-trades-feed",
  "connection_type": "sftp",
  "host": "sftp.dataprovider.com",
  "port": 22,
  "user": "cq-ingest",
  "password": "vendorPassword123",
  "prefix": "/feeds/trades/"
}

Dataset

{
  "name": "vendor-trades",
  "connection": "sftp-trades-feed",
  "data_args": {
    "file_pattern": "trades_*.csv.gz",
    "sep_override": ",",
    "encoding": "utf-8",
    "data_dt_column": ["date", "time"],
    "data_dt_format": ["%Y%m%d", "%H:%M:%S.%f"],
    "data_dt_timezone": "America/New_York",
    "data_key_column": "symbol",
    "fname_dt_regex": "trades_(\\d{8})\\.csv\\.gz",
    "fname_dt_format": "%Y%m%d",
    "fname_dt_timezone": "America/New_York",
    "fname_dt_approx_seconds": 86400,
    "arrow_sort": ["symbol", "muts"],
    "arrow_timestamp": true
  },
  "schema": [
    { "name": "symbol", "type": "string", "group": "key", "description": "Ticker symbol" },
    { "name": "date", "type": "string", "group": "time", "description": "Trade date" },
    { "name": "time", "type": "string", "group": "time", "description": "Trade time" },
    { "name": "price", "type": "double", "group": "value", "description": "Trade price" },
    { "name": "size", "type": "int64", "group": "value", "description": "Trade size" },
    { "name": "condition", "type": "string", "group": "value", "description": "Sale condition code" }
  ]
}

CloudQuant Data Liberator supports reading gzip-compressed files (.csv.gz) transparently. Use compressed files on SFTP connections to reduce transfer time over slow or high-latency links.

Network requirements

Ensure the following network connectivity from the CloudQuant Data Liberator host:

Requirement	Detail
Outbound port	TCP port 22 (or custom port) to the SFTP server
DNS resolution	The hostname must resolve from the CloudQuant Data Liberator host
Firewall rules	Whitelist the CloudQuant Data Liberator host IP on the SFTP server
SSH host key	The server’s host key must be trusted (added to known_hosts)

​SFTP

​Connection configuration

​Required fields

​Optional fields

​Example connection (password authentication)

​Example connection (key authentication)

​Dataset configuration (data_args)

​Required data_args

​Optional data_args

​Complete example

​Connection

​Dataset

​Network requirements

SFTP

Connection configuration

Required fields

Optional fields

Example connection (password authentication)

Example connection (key authentication)

Dataset configuration (data_args)

Required data_args

Optional data_args

Complete example

Connection

Dataset

Network requirements