Skip to main content

​
Advanced - Query Data Using request.post RESTful API

​
request.post from Python to Access CloudQuant Data Liberator Data

The request.post is an internet request method to request that the CloudQuant Data Liberator server accept the data enclosed arguments and process the request.

​
Post Arguments

  • 'https://api.cloudquant.ai/liberator/query' - The URL of the query interface. This must be this exact string to query.
  • data=json.dumps(...) - This json.dumps section provides the body of the query with these parameters:
    • compress
    • json_xfer
    • user
    • token
    • name
    • as_of
    • back_to
    • symbols
    • system
  • headers={'Content-Type':'application/json'}
  • cert=cert - cert comes from with pfx_to_pem('liberator.pfx','') as cert:

​
Query the Last Known Value(s)

To get the last known value in any dataset, do not provide the as_of or back_to arguments. This will give you the last known value for the given dataset and symbols.

​
Query for a Time Series Result

Adding the back_to argument for any query will give you the time series data all the way back to the specified date.
Every dataset has different data frequencies. Use short time frames until you become familiar with the datasets, as some are quite large and an over-extended request may result in an extremely large amount of data returned.

​
json.dumps Components

ArgumentDescriptionTypeExample
symbolsThe security trading symbol(s) you wish to queryString, or List"symbols":["AAPL","TLT","GOOG"]
nameThe name of the dataset (Required)String"name": "daily_bars"
as_ofThis value can be any past date so that you can see the data as it was known on the “as of” date. Defaults to now.String YYYY-MM-DD HH:MM:SS (HH:MM:SS optional)"as_of":"2020-11-22 19:51:31"
back_toThe date where the return dataset should begin.String YYYY-MM-DD HH:MM:SS (HH:MM:SS optional)"back_to":"2020-01-01"
urlOptional. The URL of the CloudQuant Data Liberator server. Defaults to https://api.cloudquant.ai.String'https://api.cloudquant.ai' or 'http://127.0.0.1:47753'
systemThe name of the authorized system from which you are querying.String"system":"API"
compressThe data compression method on the wire.BooleanTrue or False
json_xferJSON transfer. Usually False.Boolean (Always False)"json_xfer":False
userThe user identifier (as assigned by CloudQuant)String"user":"myUserID"
tokenThe user’s assigned tokenString"token":"mypersonal-private-token"

​
Example: Calling the CloudQuant Data Liberator REST API with Python

import sys
import zlib
import json
import base64
import requests
import tempfile
import contextlib
import pyarrow as pa
import OpenSSL.crypto


@contextlib.contextmanager
def pfx_to_pem(pfx_path, pfx_password):
    '''Decrypts the .pfx file to be used with requests.'''
    with tempfile.NamedTemporaryFile(suffix='.pem') as t_pem:
        t_pem.close()
        f_pem = open(t_pem.name, 'wb')
        pfx = open(pfx_path, 'rb').read()
        p12 = OpenSSL.crypto.load_pkcs12(pfx, pfx_password)
        f_pem.write(OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM, p12.get_privatekey()))
        f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, p12.get_certificate()))
        ca = p12.get_ca_certificates()
        if ca is not None:
            for cert in ca:
                f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert))
        f_pem.close()
        yield t_pem.name


compressed_transfer = True

# POST the query and prepare for a stream of single-line JSON replies
with pfx_to_pem('liberator.pfx', '') as cert:
    r = requests.post('https://api.cloudquant.ai/liberator/query',
            data=json.dumps({"compress": compressed_transfer, "json_xfer": False,
                "user": user, "token": token,
                "name": name, "as_of": as_of, "back_to": back_to, "symbols": symbols,
                "system": "API"}),
            headers={'Content-Type': 'application/json'},
            cert=cert,
            stream=True)

# Ensure that the request was successful
if r.status_code != 200:
    print(r.json())
else:
    batches = []
    for line in r.iter_lines(4096):
        # Show progress
        print('.', end='')
        sys.stdout.flush()
        # The response stream contains informational messages other than "batch"
        if line[0:14] == b'{"exception":"':
            print('Error: ' + json.loads(''.join([chr(c) for c in line]), encoding='latin-1')['value'])
            break
        if line[0:10] != b'{"batch":"':
            continue
        # Cut out the payload from the JSON line without json module overhead
        decoded = base64.b64decode(line[10:-3])
        # Interpret the payload as an Apache Arrow IPC stream
        reader = pa.ipc.open_stream(zlib.decompress(decoded) if compressed_transfer else decoded)
        # Accumulate RecordBatch objects in a list
        batches.extend([batch for batch in reader])
    print('')
    if batches:
        # Create an Arrow Table view on the batches and convert to a pandas DataFrame
        df = pa.Table.from_batches(batches).to_pandas()
        if 'muts' in df and '_seq' in df:
            # By default, the data is sorted by symbol... re-sort by time
            df = df.sort_values(['muts', '_seq'])
            df.reset_index(drop=True, inplace=True)
        print(df)