The following examples are coded for Linux bash scripts. They require:
- curl
- jq
- base64
- python with pyarrow
- pigz
Parameters that Can Be Added to /liberator/datasets
|
Argument |
Description |
Type |
Example |
|
symbols |
The security trading symbol(s) you wish to query |
String or list of string |
"symbols":["AAPL","IBM"] |
|
name |
The name of the dataset |
string |
"name":"daily_bars" |
|
as _of |
This value can be any past date so that you can see the data as it was known on the “as of” date. as_of defaults to the current time. |
string Format YYYY-MM-DD HH:MM:SS (HH:MM:SS optional) |
"as_of":"2023-01-23" Or "as_of":"2023-01-23 23:55"
|
|
back_to |
The date where the return dataset should begin. This is reading all the data “back to” the specified date. |
string Format YYYY-MM-DD HH:MM:SS (HH:MM:SS optional) |
"back_to":"2023-01-23" Or "back_to":"2023-01-23 23:55"
|
|
fields |
An optional filter of field names. (There are some mandatory fields) |
String or list of strings |
"fields":["high", "low", "volume"] |
|
stats |
Set to 'total' to get count per symbol as Json result |
string |
“stats”, “total” |
|
crux_key |
if querying for a Crux data set |
string |
“crux_key”, “<Your Key>” |
|
compress |
The data compression method on the wire. CloudQuant uses compression. |
Boolean |
“compress", false |
|
json_xfer |
Json transfer. Almost always true for Restful clients. |
Boolean |
"json_xfer":true |
|
record_limit |
Limit number of records |
int |
“record_limit”, 5 |
|
user |
The user identifier (as assigned by CloudQuant) |
string |
“user”,”myUserID” |
|
token |
The user’s assigned token |
string |
“token”,”mypersonal-private-token” |
|
debug_stream |
return the debug stream |
Boolean or string |
“debug_stream”:false |
|
warning_stream |
return the warning stream |
Boolean or string |
“warning_stream”:false |
Query the Last Known Value
Example: query_current_value.sh – Retrieve the Current Value for given symbol(s)
Example using CURL to query the current or last known value for the given symbols from a given dataset.
#!/bin/bash
URL=https://api.cloudquant.ai
USER=TODO-ADD-YOUR-USERID-HERE
TOKEN=TODO-ADD-YOUR-TOKEN-HERE
curl -qs --cert-type P12 --cert liberator.pfx -H "Content-Type: application/json" --noproxy '*' --data '{"json_xfer":true,"symbols":["AAPL","IBM"],"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API", "debug_stream":false}' $URL/liberator/query
Output
The output will come as arrays of JSON objects.
[
[{"_seq":4737,"_dsname":"bar","timestamp":"2023-01-24 20:00:00.000","msg_len":74,"msg":"b","muts":1674608400000000,"symbol":"IBM","length":23400,"open":141.25,"open_primary":0.0,"high":142.75,"low":140.0,"close":141.4900054932,"close_primary":141.4900054932,"volume":4407460,"vwap":141.6482696533,"spread":0.0301483665,"bidvol":1222215,"askvol":970141,"count":58440},{"_seq":26,"_dsname":"bar","timestamp":"2023-01-24 20:00:00.000","msg_len":74,"msg":"b","muts":1674608400000000,"symbol":"AAPL","length":23400,"open":140.3049926758,"open_primary":140.4700012207,"high":143.1600036621,"low":140.3000030518,"close":142.5299987793,"close_primary":142.5299987793,"volume":66430864,"vwap":142.1145172119,"spread":0.0122285504,"bidvol":20850247,"askvol":15903883,"count":498637}]
]
In this case you can see JSON objects for both IBM and AAPL
• IBM
{"_seq":4737,"_dsname":"bar","timestamp":"2023-01-24 20:00:00.000","msg_len":74,"msg":"b","muts":1674608400000000,"symbol":"IBM","length":23400,"open":141.25,"open_primary":0.0,"high":142.75,"low":140.0,"close":141.4900054932,"close_primary":141.4900054932,"volume":4407460,"vwap":141.6482696533,"spread":0.0301483665,"bidvol":1222215,"askvol":970141,"count":58440}
• AAPL
{"_seq":26,"_dsname":"bar","timestamp":"2023-01-24 20:00:00.000","msg_len":74,"msg":"b","muts":1674608400000000,"symbol":"AAPL","length":23400,"open":140.3049926758,"open_primary":140.4700012207,"high":143.1600036621,"low":140.3000030518,"close":142.5299987793,"close_primary":142.5299987793,"volume":66430864,"vwap":142.1145172119,"spread":0.0122285504,"bidvol":20850247,"askvol":15903883,"count":498637}
Query a Time Range (Ending in the Current Time)
Example: query_time_range.sh – Retrieve the time range of data for given symbol(s)
#!/bin/bash
URL=https://api.cloudquant.ai
USER=TODO-ADD-YOUR-USERID-HERE
TOKEN=TODO-ADD-YOUR-TOKEN-HERE
# query time range of data for a single symbol
curl -qs --cert-type P12 --cert liberator.pfx -H "Content-Type: application/json" --noproxy '*' --data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API", "back_to":"2022-12-01", "debug_stream":false}' $URL/liberator/query
Output
The output will come as arrays of JSON objects.
[{"heartbeat":""},
{"heartbeat":""},
{"heartbeat":""},
{"heartbeat":""},
{"heartbeat":""},
[{"_seq":25,"_dsname":"bar","timestamp":"2023-01-03 20:00:00.000","msg_len":74,"msg":"b","muts":1672794000000000,"symbol":"AAPL","length":23400,"open":130.2799987793,"open_primary":130.2100067139,"high":130.8999938965,"low":124.1699981689,"close":125.0699996948,"close_primary":125.0699996948,"volume":112136343,"vwap":125.6752548218,"spread":0.0117298001,"bidvol":41324933,"askvol":31958667,"count":1021024},{"_seq":26,"_dsname":"bar","timestamp":"2023-01-04 20:00:00.000","msg_len":74,"msg":"b","muts":1672880400000000,"symbol":"AAPL","length":23400,"open":126.8899993896,"open_primary":127.0,"high":128.6557006836,"low":125.0800018311,"close":126.3600006104,"close_primary":126.3600006104,"volume":89116019,"vwap":126.6606826782,"spread":0.0134841939,"bidvol":27480262,"askvol":30708395,"count":770032},{"_seq":25,"_dsname":"bar","timestamp":"2023-01-05 20:00:00.000","msg_len":74,"msg":"b","muts":1672966800000000,"symbol":"AAPL","length":23400,"open":127.1299972534,"open_primary":127.0699996948,"high":127.7699966431,"low":124.7600021362,"close":125.0199966431,"close_primary":125.0199966431,"volume":81200778,"vwap":126.1032333374,"spread":0.0123035843,"bidvol":23206926,"askvol":28996089,"count":665425},{"_seq":25,"_dsname":"bar","timestamp":"2023-01-06 20:00:00.000","msg_len":74,"msg":"b","muts":1673053200000000,"symbol":"AAPL","length":23400,"open":126.0100021362,"open_primary":126.0,"high":130.2899932861,"low":124.8899993896,"close":129.6199951172,"close_primary":129.6199951172,"volume":87748469,"vwap":128.2108764648,"spread":0.0118012521,"bidvol":33240574,"askvol":24168704,"count":711475},{"_seq":26,"_dsname":"bar","timestamp":"2023-01-09 20:00:00.000","msg_len":74,"msg":"b","muts":1673312400000000,"symbol":"AAPL","length":23400,"open":130.4649963379,"open_primary":130.5299987793,"high":133.4100036621,"low":129.8899993896,"close":130.1499938965,"close_primary":130.1499938965,"volume":70843060,"vwap":131.7045288086,"spread":0.0109032113,"bidvol":19911462,"askvol":25075210,"count":645334},{"_seq":25,"_dsname":"bar","timestamp":"2023-01-10 20:00:00.000","msg_len":74,"msg":"b","muts":1673398800000000,"symbol":"AAPL","length":23400,"open":130.2599945068,"open_primary":130.25,"high":131.2635955811,"low":128.1199951172,"close":130.7299957275,"close_primary":130.7299957275,"volume":63895545,"vwap":129.798828125,"spread":0.0114623047,"bidvol":18644164,"askvol":23521748,"count":554916},{"_seq":23,"_dsname":"bar","timestamp":"2023-01-11 20:00:00.000","msg_len":74,"msg":"b","muts":1673485200000000,"symbol":"AAPL","length":23400,"open":131.25,"open_primary":131.25,"high":133.5099945068,"low":130.4600067139,"close":133.4900054932,"close_primary":133.4900054932,"volume":69502538,"vwap":132.2747802734,"spread":0.0113208676,"bidvol":25598436,"askvol":19814992,"count":561097},{"_seq":27,"_dsname":"bar","timestamp":"2023-01-12 20:00:00.000","msg_len":74,"msg":"b","muts":1673571600000000,"symbol":"AAPL","length":23400,"open":133.8800048828,"open_primary":133.9799957275,"high":134.2599945068,"low":131.4400024414,"close":133.4100036621,"close_primary":133.4100036621,"volume":71378618,"vwap":133.1491241455,"spread":0.0124849975,"bidvol":24132872,"askvol":19385490,"count":635291},{"_seq":26,"_dsname":"bar","timestamp":"2023-01-13 20:00:00.000","msg_len":74,"msg":"b","muts":1673658000000000,"symbol":"AAPL","length":23400,"open":132.0299987793,"open_primary":132.0,"high":134.9199981689,"low":131.6600036621,"close":134.7599945068,"close_primary":134.7599945068,"volume":57809365,"vwap":133.6643981934,"spread":0.0113926539,"bidvol":21034691,"askvol":16577038,"count":537332},{"_seq":26,"_dsname":"bar","timestamp":"2023-01-17 20:00:00.000","msg_len":74,"msg":"b","muts":1674003600000000,"symbol":"AAPL","length":23400,"open":134.8300018311,"open_primary":134.7599945068,"high":137.2899932861,"low":134.1300048828,"close":135.9400024414,"close_primary":135.9400024414,"volume":63643042,"vwap":135.7635803223,"spread":0.0121124163,"bidvol":22644974,"askvol":17963464,"count":595790},{"_seq":26,"_dsname":"bar","timestamp":"2023-01-18 20:00:00.000","msg_len":74,"msg":"b","muts":1674090000000000,"symbol":"AAPL","length":23400,"open":136.8150024414,"open_primary":136.8399963379,"high":138.6100006104,"low":135.0299987793,"close":135.2100067139,"close_primary":135.2100067139,"volume":69685283,"vwap":136.387512207,"spread":0.0119594801,"bidvol":24333322,"askvol":20330802,"count":578246},{"_seq":24,"_dsname":"bar","timestamp":"2023-01-19 20:00:00.000","msg_len":74,"msg":"b","muts":1674176400000000,"symbol":"AAPL","length":23400,"open":134.0800018311,"open_primary":134.0,"high":136.25,"low":133.7700042725,"close":135.2700042725,"close_primary":135.2700042725,"volume":58296827,"vwap":134.973236084,"spread":0.012528941,"bidvol":16422539,"askvol":21399785,"count":491645},{"_seq":25,"_dsname":"bar","timestamp":"2023-01-20 20:00:00.000","msg_len":74,"msg":"b","muts":1674262800000000,"symbol":"AAPL","length":23400,"open":135.2799987793,"open_primary":135.1199951172,"high":138.0200042725,"low":134.2200012207,"close":137.8699951172,"close_primary":137.8699951172,"volume":80221643,"vwap":136.2800445557,"spread":0.0113122445,"bidvol":35391858,"askvol":18906684,"count":552190},{"_seq":26,"_dsname":"bar","timestamp":"2023-01-23 20:00:00.000","msg_len":74,"msg":"b","muts":1674522000000000,"symbol":"AAPL","length":23400,"open":138.1199951172,"open_primary":138.1300048828,"high":143.3150024414,"low":137.8999938965,"close":141.1100006104,"close_primary":141.1100006104,"volume":81982177,"vwap":141.2467041016,"spread":0.0116401315,"bidvol":30546096,"askvol":23863977,"count":719269},{"_seq":26,"_dsname":"bar","timestamp":"2023-01-24 20:00:00.000","msg_len":74,"msg":"b","muts":1674608400000000,"symbol":"AAPL","length":23400,"open":140.3049926758,"open_primary":140.4700012207,"high":143.1600036621,"low":140.3000030518,"close":142.5299987793,"close_primary":142.5299987793,"volume":66430864,"vwap":142.1145172119,"spread":0.0122285504,"bidvol":20850247,"askvol":15903883,"count":498637}]
]
Query a Point-in-Time
Example: query_point_in_time.sh
Retrieving the last known value at a given point-in-time in the past.
#!/bin/bash
URL=https://api.cloudquant.ai
USER=TODO-ADD-YOUR-USERID-HERE
TOKEN=TODO-ADD-YOUR-TOKEN-HERE
# query point-in-time data for a single symbol on a given as_of date
curl -qs --cert-type P12 --cert liberator.pfx -H "Content-Type: application/json" --noproxy '*' --data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API", "as_of":"2022-12-15", "debug_stream":false}' $URL/liberator/query
Output
[
[{"_seq":26,"_dsname":"bar","timestamp":"2022-12-14 20:00:00.000","msg_len":74,"msg":"b","muts":1671066000000000,"symbol":"AAPL","length":23400,"open":145.3500061035,"open_primary":145.3200073242,"high":146.6549987793,"low":141.1600036621,"close":143.2100067139,"close_primary":143.2100067139,"volume":82491309,"vwap":144.3030090332,"spread":0.0181106683,"bidvol":21701396,"askvol":32508895,"count":597855}]
]
This output is the result of a request to get the data for AAPL on 2022-12-15. The data timestamp is 2022-12-14 20:00:00.000. This was the previous close before 2022-12-15 00:00:00 and was the last known value at that given time.
Other Query Examples
The following code can be found in the other_queries.sh file in your download for curl api.
# NOTE: JSON format will be changing soon to accommodate streaming
curl -qs --cert-type P12 --cert liberator.pfx -H "Content-Type: application/json" --noproxy '*' --data '{"json_xfer":true,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' $URL/liberator/query
# certain symbols
curl -qs --cert-type P12 --cert liberator.pfx -H "Content-Type: application/json" --noproxy '*' --data '{"json_xfer":true,"symbols":["AAPL","IBM"],"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' $URL/liberator/query
# turn off log messages
curl -qs --cert-type P12 --cert liberator.pfx -H "Content-Type: application/json" --noproxy '*' --data '{"json_xfer":true,"symbols":["AAPL","IBM"],"debug_stream":false,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' $URL/liberator/query
# date range in America/New_York ... [back_to,as_of) ... as_of=now if not specified
curl -qs --cert-type P12 --cert liberator.pfx -H "Content-Type: application/json" --noproxy '*' --data '{"json_xfer":true,"symbols":["AAPL","IBM"],"back_to":"2023-01-01 00:00:00","debug_stream":false,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' $URL/liberator/query
# back_to/as_of can be microseconds since 1970 UTC
curl -qs --cert-type P12 --cert liberator.pfx -H "Content-Type: application/json" --noproxy '*' --data '{"json_xfer":true,"symbols":["AAPL","IBM"],"back_to":1672552800000000,"as_of":1674577386000000,"debug_stream":false,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' $URL/liberator/query
# NOTE: Apache Arrow is the preferred transfer mechanism (json_xfer=false)
# get base64-encoded batches, compression off
BATCHES=$(curl -qs --cert-type P12 --cert liberator.pfx -H "Content-Type: application/json" --noproxy '*' --data '{"json_xfer":false,"compress":false,"symbols":["AAPL"],"back_to":"2023-01-01","debug_stream":false,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' $URL/liberator/query | jq -r '.[] | select(type=="object") | .batch')
for BATCH in $BATCHES; do
echo $BATCH | base64 -d | python -c "import sys; import pyarrow as pa; print(pa.ipc.open_stream(sys.stdin.buffer.read()).read_next_batch().to_pylist())"
done
# get base64-encoded batches, compression on
BATCHES=$(curl -qs --cert-type P12 --cert liberator.pfx -H "Content-Type: application/json" --noproxy '*' --data '{"json_xfer":false,"compress":true,"symbols":["AAPL"],"back_to":"2023-01-01","debug_stream":false,"name":"daily_bars","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' $URL/liberator/query | jq -r '.[] | select(type=="object") | .batch')
for BATCH in $BATCHES; do
echo $BATCH | base64 -d | pigz -dz | python -c "import sys; import pyarrow as pa; print(pa.ipc.open_stream(sys.stdin.buffer.read()).read_next_batch().to_pylist())"
done