Queries & working with large datasets

The CloudQuant Data Liberator service frequently returns extensive result sets, sometimes reaching millions of rows. These queries can be time-consuming to execute. Since data frequencies vary across datasets, start with narrow timeframes for specific symbols before expanding scope.

Consider running a point-in-time query for one symbol to get an idea of how large your dataset is prior to running other queries.

As of release version 2.0, the P12 client certificate (liberator.pfx) is no longer required. The --cert-type P12 --cert liberator.pfx flags shown in the curl examples below are only needed for releases prior to 2.0.

Query types

Most dataset queries fall into two categories:

Point-in-time queries
Time series queries

Familiarity with both query types early on will improve your API usage efficiency.

Point-in-time queries

Omit the back_to parameter to receive single point-in-time data for each specified symbol based on the as_of date/time. If you also exclude as_of, CloudQuant Data Liberator defaults to the current date/time.

df = liberator.get_dataframe(
    liberator.query(
        name='daily_bars',
        as_of='2020-11-15',
        symbols=['AAPL']
    )
)

var res = liberator.query(new Dictionary<string, dynamic>() {
    {"name", "daily_bars"},
    {"as_of", "2020-11-15"},
    {"symbols", "AAPL"}
});

let params = {
    symbols: "AAPL",
    name:    "daily_bars",
    as_of:   "2020-11-15"
};

liberator.query(params).then(function(results) {
    // Your code
});

Object query_id = liberator.query(new HashMap<>() {{
    put("name", "daily_bars");
    put("as_of", "2020-11-15");
    put("symbols", "AAPL");
}});

print(liberator::get_dataframe(liberator::query(
    name="daily_bars",
    as_of="2020-11-15",
    symbols="AAPL"
)))

curl -qs --cert-type P12 --cert liberator.pfx \
  -H "Content-Type: application/json" --noproxy '*' \
  --data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","as_of":"2020-11-15","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
  $URL/liberator/query

auto table = liberator.get_table(liberator.query({
    {"symbols", std::vector<std::string>{"AAPL"}},
    {"name", "daily_bars"s},
    {"as_of", "2020-11-15"s}
}));

Time series queries

Include a back_to parameter that precedes your as_of value. When as_of is excluded, it defaults to the present moment.

df = liberator.get_dataframe(
    liberator.query(
        name='daily_bars',
        as_of='2020-11-15',
        back_to='2018-11-15',
        symbols=['FB', 'AAPL', 'NFLX', 'GOOG', 'MSFT', 'IBM']
    )
)

var res = liberator.query(new Dictionary<string, dynamic>() {
    {"name", "daily_bars"},
    {"as_of", "2021-04-10"},
    {"back_to", "2021-03-10"},
    {"symbols", "ES"}
});

foreach (RecordBatch batch in res())
{
    var df = DataFrame.FromArrowRecordBatch(batch);
    Console.WriteLine(DataFrameUtils.PrettyPrint(df));
}

let params = {
    back_to: "2021-01-12 00:00:00",
    as_of:   "2021-01-12 23:59:59",
    symbols: "AAPL",
    name:    "daily_bars"
};

liberator.query(params).then(function(results) {
    // Your code
});

Object query_id = liberator.query(new HashMap<>() {{
    put("name", "daily_bars");
    put("as_of", "2020-11-15");
    put("back_to", "2020-01-01");
    put("symbols", "AAPL");
}});

print(liberator::get_dataframe(liberator::query(
    name="daily_bars",
    symbols="AAPL",
    as_of="2019-09-15",
    back_to="2019-09-10"
)))

curl -qs --cert-type P12 --cert liberator.pfx \
  -H "Content-Type: application/json" --noproxy '*' \
  --data '{"json_xfer":true,"symbols":"AAPL","name":"daily_bars","back_to":"2022-12-01","user":"'"$USER"'","token":"'"$TOKEN"'","system":"API"}' \
  $URL/liberator/query

auto table = liberator.get_table(liberator.query({
    {"symbols", std::vector<std::string>{"AAPL", "GOOGL"}},
    {"name", "daily_bars"s},
    {"as_of", "2021-04-10"s},
    {"back_to", "2021-03-10"s}
}));

When using back_to or as_of parameters, the time component is always used even if you do not specify it. Therefore, if you say as_of: "2023-01-15", you are actually saying as_of: "2023-01-15 00:00:00". This may affect result precision depending on your data requirements.

⌘I

Concepts

Python

C#

JavaScript

Java

R Language

RESTful API

C++

Excel Plug-In

Queries & Working with Large Datasets

Queries & working with large datasets

Query types

Point-in-time queries

Time series queries

​Queries & working with large datasets

​Query types

​Point-in-time queries

​Time series queries

Queries & working with large datasets

Query types

Point-in-time queries

Time series queries