CQAI - Liberator API for Python Users in CloudQuant.AI

CloudQuant AI is the hosted jupyter notebook environment for data research. Analysts utilize this environment to review alternative data.

Required Imports

import liberator

Useful Imports

The following imports, while not actually require, are very useful in programming calls and processing results from liberator.

import json

from liberator.util import *

from datetime import datetime as dt

import datetime

from datetime import timedelta

Listing Available Datasets

Example

res = liberator.datasets()

print(res)

The liberator.datasets() call will return a dictionary object with all the datasets available. The above code prints out a return similar to the following.

{

'Intraday': ['SSR_Forward_DSI', 'Halt_Production_DSI', 'SSR_Production_DSI', 'News', 'Twitter', 'Stocktwits', 'Halt_Forward_DSI', 'spiderrock_printsets'],
'Monthly': ['Joblink Ticker Mapping', 'VerticalKnowledge Indeed'],
'Weekly': ['VerticalKnowledge AmazonJobs'],
'Daily': ['DXOpen IM', 'Float Shares', 'Fundamental', 'Share Stats', 'Valuengine Model', 'GSQ ESG Large Cap, Percentile Ranks', 'Brain BSR 2-Day Alpha', 'Brain BSR 5-Day Alpha', 'Brain BSR 21-Day Alpha', 'Brain BSR Stock Universe', 'CruxTest', 'Brain BSI Stock List', 'Brain BLMCF Stock List', 'Likefolio Unadjusted', 'Likefolio Adjusted', 'Tesseract', 'Catalog', 'Insight Summary', 'Bombora Daily Detail', 'Joblink Posting Details', 'Danel SmartUS', 'Danel SmartUS Adj', 'ETFGlobal Constituents', 'DXOpen SF', 'Joblink Job Activity', 'Joblink Job Details', 'Intelligration Vestly', 'Bombora Country-Domain-State-Topic', 'EDI Security Master', 'PrecisionAlpha XNYS', 'GSQ ESG Large Cap, Short Term Price Predictor', 'Bombora Daily Parsed', 'GSQ ESG MidSmall Cap, Measure Outcome Values', 'DXOpen SF Amer', 'DXOpen SF Asia', 'DXOpen SF Euro', 'DXOpen SF Other', 'DXOpen IM Amer', 'DXOpen IM Asia', 'DXOpen IM Euro', 'DXOpen IM Other', 'EDI Corporate Actions', 'GSQ ESG MidSmall Cap, Percentile Ranks', 'GSQ ESG Large Cap, Measure Outcome Values', 'Brain BSI 7-Day Sentiment', 'PrecisionAlpha XNAS', 'Brain BSI 30-Day Sentiment', 'Brain BLMCF 10-K Differences', 'Brain BLMCF All Differences', 'Brain BLMCF All Metrics', 'Brain BLMCF 10-K Metrics', 'S3 Partners', 'GSQ ESG MidSmall Cap, Short Term Price Predictor', 'ETFGlobal Industry', 'ETFGlobal Constituents Unaltered', 'ETFGlobal Analytics', 'ETFGlobal Fundflow', 'Symbol Directory', 'spiderrock_livesurfaceterm_eod', 'spiderrock_livesurfaceterm_intraday'],
'Market Data': ['Daily Bars', 'Minute Bars', 'NBBO', 'Trades', 'Daily Bars Adjusted', 'Minute Bars Raw']

}

You can make this output easier to understand by passing the results into json.dumps() as shown below to get a better output.

print(json.dumps(res,indent=4))

 

{

"Intraday": [

"SSR_Forward_DSI",

"Halt_Production_DSI",

"SSR_Production_DSI",

"News",

"Twitter",

"Stocktwits",

"Halt_Forward_DSI",

"spiderrock_printsets"

],

"Monthly": [

"Joblink Ticker Mapping",

"VerticalKnowledge Indeed"

],

"Weekly": [

…

"Market Data": [

"Daily Bars",

"Minute Bars",

"NBBO",

"Trades",

"Daily Bars Adjusted",

"Minute Bars Raw"

]

}

Query a Named Dataset

The query method allows you to read a named dataset. The name of the dataset must equal one of the names that is found in the liberator.datasets() result set.

Example:

res = liberator.query( name='Stocktwits')

 

Return results are in the form of a python generator. Use the get_dataframe() method to translate the data into a pandas dataframe.

Common Keyword Arguments used in liberator.query()

Argument	Description	Type	Example
symbols	The security trading symbol(s) you wish to query	string, or list	symbols = “AAPL” symbols = [“AAPL”, “NFLX”]
name	The name of the dataset	String	name=”Daily Bars”
as _of	The date in time that you wish the data to be. as_of defaults to now. This value can be any past date so that you can see the data as it was known on the “as of” date.	String Format YYYY-MM-DD HH:MM:SS   HH:MM:SS is optional	as_of = “2020-11-22 19:51:31”
back_to	The date where the return dataset should begin. This is reading all the data “back to” the specified date.	String Format YYYY-MM-DD HH:MM:SS   HH:MM:SS is optional	back_to = “2020-01-01”

 

Query the Last Known Value(s)

To get the last known value in any dataset simply do not provide the as_of or back_to arguments. This will give you the last known value for the given dataset and symbols.

Query for a Time Series Result

Adding the back_to argument for any query will give you the time series data all the way back to the specified back to date.

Every dataset has different data frequencies. Some update very infrequently. For example, the earnings reports for a stock only update every 3 months. Others update very frequently intraday. A query that ask for a time series may still result in one or no results for the given symbol.

 

Working with Dataframes: Liberator.get_dataframe()

The liberator API has a function that converts the (generator) results received from the query() into a dataframe.

Example:

df = Liberator.get_dataframe(res)

Liberator API for External API using Python

The liberator service is a restful API allowing authenticated users access to the hosted data sets.

Prerequisites

• Python 3

• requests module must be installed

• pyarrow module must be installed

• os module installed (usually part of python 3)

Authenticating The API User

When you receive the onboarding email from CloudQuant Customer Success will contain 3 files.

• cq.crt – your personal certificate for access liberator

• cq.key - your personal key file used to access liberator

• example.py – python example code

Query using request.post

The request.post is an internet request method to request that the liberator server accept the data enclosed arguments and process the request.

Post Arguments:

• 'https://api.cloudquant.ai/liberator/query'

  • This is the url of the query interface. This must be this exact string to query.

• data=json.dumps({"compress":compressed_transfer,"json_xfer":False, "user":user,"token":token, "name":name,"as_of":as_of,"back_to":back_to,"symbols":symbols,"system":"API"})

  • This json.dumps section provides the body of the query. As in the CQ AI we specify the same values to the query.

• headers={'Content-Type':'application/json'},

• cert=('cq.crt','cq.key'),

• stream=True

Argument	Description	Type	Example
symbols	The security trading symbol(s) you wish to query	string, or list	"symbols":[“AAPL”,”TLT”,”GOOG”]
name	The name of the dataset	String	“name”:Daily Bars”
as _of	The date in time that you wish the data to be. as_of defaults to now. This value can be any past date so that you can see the data as it was known on the “as of” date.	String Format YYYY-MM-DD HH:MM:SS   HH:MM:SS is optional	“as_of”:“2020-11-22 19:51:31”
back_to	The date where the return dataset should begin. This is reading all the data “back to” the specified date.	String Format YYYY-MM-DD HH:MM:SS   HH:MM:SS is optional	“back_to”:“2020-01-01”
system	The name of the authorized system that you are querying from. The security mechanism in liberator authorized users for individual systems.	String   This is almost always “API”	“system”:“API”
compress	The data compression method on the wire. CloudQuant uses compression.	String. Always compressed_transfer	"compress":compressed_transfer
json_xfer	Json transfer. This is usually False	Boolean. Always False	"json_xfer":False,
user	The user identifier (as assigned by CloudQuant)	String	"user":”myUserID”
token	The user’s assigned token	String	"token":”mypersonal-private-token”

 

Query the Last Known Value(s)

To get the last known value in any dataset simply do not provide the as_of or back_to arguments. This will give you the last known value for the given dataset and symbols.

Query for a Time Series Result

Adding the back_to argument for any query will give you the time series data all the way back to the specified back to date.

Every dataset has different data frequencies. Some update very infrequently. For example, the earnings reports for a stock only update every 3 months. Others update very frequently intraday. A query that ask for a time series may still result in one or no results for the given symbol.

Example Calling the Liberator API with Python

Example Code for calling query interface in liberator using POST

import zlib

import json

import base64

import requests

import pyarrow as pa

compressed_transfer=True

# POST the query and prepare for a stream of single-line JSON replies

r=requests.post('https://api.cloudquant.ai/liberator/query',

data=json.dumps({"compress":compressed_transfer,"json_xfer":False,

"user":user,"token":token,

"name":name,"as_of":as_of,"back_to":back_to,"symbols":symbols,

"system":"API"}),

headers={'Content-Type':'application/json'},

cert=('cq.crt','cq.key'),

stream=True)

# ensure that the request was successful

if r.status_code!=200:

print(r.json())

else:

batches=[]

for line in r.iter_lines(4096):

# The response stream contains informational messages other than "batch"

if line[0:10] != b'{"batch":"':

continue

# cut out the payload from the JSON line w/o json module overhead

decoded=base64.b64decode(line[10:-3]) # "},

# interpret the payload as an Apache Arrow IPC stream

reader=pa.ipc.open_stream(zlib.decompress(decoded) if compressed_transfer else decoded)

# accumulate RecordBatch objects in a list

batches.extend([batch for batch in reader])

# create an Arrow Table view on the batches and convert (e.g.) to a pandas DataFrame

df=pa.Table.from_batches(batches).to_pandas()

if 'muts' in df and '_seq' in df:

# by default, the data is sorted by symbol... re-sort by time

df=df.sort_values(['muts', '_seq'])

df.reset_index(drop=True,inplace=True)

print(df)