Extending Qlik

Working with Qlik Sense Engine protocol: four problems you need to solve

Qlik Sense has several APIs and a .Net SDK to help you work with the QIX Engine. They help you working with Qlik Sense without having to bother about the details. But if you need to work on a lower level, there are some problems you need to solve.

1. Connect over Web Socket

Qlik Sense QIX engine talks Web Socket, and only web socket. While you might need to make a https call to trigger authentication, the actual communication with the engine is web socket only.

The format of the URL can be seen in the Engine API Explorer, which you will find in Qlik Sense devhub:

ws://localhost:4848/app/c%3A%5Cusers%5Cerikw%5Cdocuments%5Cqlik%5Csense%5Capps%5Cexecutive%20dashboard.qvf?

Note that the app id is included in the URL, but what you get is an open web socket, not an open document. Actually, in Qlik Sense desktop it will work without the app id in the URL. That’s because the app id in the URL is not for QIX Engine. It’s for the proxy you will go through to reach Engine, and used for load balancing, the proxy uses it to find which engine to use for this app. This also means that you should always open a new web socket when you open a new app, since you need to give the proxy a chance to select the correct engine.

Conclusion:

you need to use web sockets to talk to QIX Engine
the web socket URL should contain the app id
you should always open a new socket for each app

2. Match request and response

With traditional HTTP communication you make a request, wait, and get a response. Web sockets work differently. You send a message to QIX Engine, but don’t wait for the answer. You just keep on sending messages, and eventually QIX Engine will reply. You do not know the order of the responses and actually the protocol does not tell you what respone belongs to which request. Typically it looks something like this:

This is the initial communication when Qlik Sense standard client opens an app. If you think it is hard to read (and it is) you can open the browser developer tools and look in the network tab, the image above is from Chrome.

The darker rows are requests sent from the client to QIX Engine, while the white rows are responses. As you can se the client makes a series of requests (seven) and after that comes the respones. It is not obvious which responses belong to which call, and the respones can not be understood without knowing the request.

So how do you know which respons belongs to which request? Qlik Sense uses a standard called JSON-RPC for this. This means that in each request there is a field called id that contains a unique identifier for this request (in this case just a number 1, 2, 3 etc). The reply for the call contains the same id. So if you are going to build your own protocol handler you need to save the ids and use them to connect request and reply.

But note the very first line above, it has no id. Thats because there is no request for it, it is what JSON-RPC calls a notification.

Conclusion:

web sockets are not like traditional HTTP in that you do not wait for a response for your request
instead requests and responses are connected with an id included in the packages
you might get messages without id, in that case they are notifications

3. Handle the handles

A key concept in the QIX engine protocol is the handle. For every object you open or create you get a number, a handle. Do not confuse that with the request id (also a number) or the object id (a string, in most cases a short or long GUID). You need to save the handle when you open or create an object and use it for subsequent calls.

Initially you have only one handle, the global handle, which is -1. When you open your app you get the app handle, in most cases 1.

An example from the list above:

Conclusion:

QIX engine uses something called handle to keep track of objects
you always need a handle in your calls to QIX Engine
the handle will be in the response from calls that create or open an object

4. Validate when changed

Some responses from the QIX engine contains another field, ‘change’. This field is an array of numbers. The numbers are actually handles and refers to objects that have been invalidated by the call. If the user for example makes a selection, she will pretty quickly get a response. This response means that QIX engine has received the selection.

It has also determined which objects of the ones you have opened that are affected (invalidated is often the term used) by this call. This means that the data you have for these objects is no longer valid. It is your responsibility to fetch the new data, QIX engine will not automatically recalculate it for you, only tell you that the previous version is no longer valid.

For some calls there will be no change array, while for some selections(or clear selections) a lot of open objects will be affected, so the array will be long. The apis automatically refresh the data for you, but if you are building your own stuff you need to take care of it yourself. You do that with the getLayout call for the object.

If you do not refresh your data, QIX will not tell you again that it is invalid. You only get notified once, until, of course, you get new data and that is invalidated.

Conclusion:

QIX engine keeps track of which open objects are affected by a call
the field change contains an array of these objects
QIX will not sendd the new data until you asks for it
it is your responsibility to keep track of the state of the data you have fetched from the QIX engine

Qlik Sense Generic Object IV: The lists

A Qlik Sense Generic Object can also be used to list objects in the Qlik Sense app to which it belongs. You can use it to list things like:

fields
dimensions
measures
bookmarks
snapshots
media (that is images controlled by Qlik Sense)
sheets
master objects
variables
stories

The javascript APIs provide the app.getList method, which is a wrapper that helps you get theses lists. It will create a session object for you with the list you are requesting, get that actual data in the list and call your callback function once the data is available.

List are live – your callbacks might be called several times

Just like with other Generic Objects the data might be invalidated. If you are using the javascript APIs it will then be automatically revalidated and your callback function will be called again, so you should be prepared that your callback might be called multiple times. But unlike the Hypercube and List Object the lists are not affected by the users selections. Instead they will be invalidated when the underlying data changes:

if the user adds a bookmark, you will get a new bookmarklist
if the user adds a measure, you will get a new measurelist
if the user adds a dimension, you will get a new dimensionlist
etc….

Configure the data you get in your list

The data you get in your lists is configurable. If you use the API method, you will get a default set of fields for each item in the list, but you can set the list up to suite your needs.

As an example, if you use the api method getList(‘measure’, callback), you will get a list where records look like this:

The first and second part, qInfo and qMeta, will always be there. They contain the id to use when refering to this measure and title and description use ful in a list of measures. The third part however, qData, is configurable. The default configuration just gives you tags and title one more time, but if you instead want more data about the measure, you can use your own measure list definition like this:

Under qData you provide your own mapping of the properties you want. In this case I have just specified that I want everything under qMeasure included. You can also specify individual fields.

And get output that includes more data for the measures:

And the same goes for the other lists, you can supply your own definitions and get more data included.

Qlik Sense Generic Object III: Listboxes

SWhile the HyperCube is the main object for calculating data the ListboxObject is a way to list available values for a field. Its the object behind the listboxes in the Filter Panel. It is also the object used for the listboxes that are created when you click on a field in the selection toolbar.

An example setup from the horizontal listbox extension example, included with the Qlik Sense installation:

This contains defaults for most of the parameters, except the actual field, which the user selects in the property panel, which looks like this:

As you can see the Listbox has either a predefined measure from the library, in which case qListObjectDef.qLibraryId will contain the id of the measure (some UID string) or it has a field name, which will be in the array qListObjectDef.qDef.qFieldDefs at the first position. You don’t really have to bother about which one the user selects to use, the ListboxObject will work the same anyhow.

Frequencies and sort order

There are also some other properties in the ListboxObject that are well known to QlikView users. The frequence mode gives you some additional info on the frequency of the respective values in the ListboxObject. Look to horizontal listbox extension for an example. There is also the qSortCriterias array, which allows you to specify the sort order. The default is to have the selected values first, but you might want to for example keep months order from January to December irrespective of which month is selected.

Mashup editor – even more properties

Like with the Hypercube, you can also build a ListboxObject in the mashup editor. The tool to do this looks like this:

As you can see, you also get the option to include expressions in your listbox, usable if you want to make some calculations for each value in the list. The mashup editor will create a ListboxObject for you like this:

The expressions are added to the ListboxObject in the qExpressions array. In this example I have used predefined measures, but you can also use measures defined in the actual listbox object. The mashup editor includes a expression editor (actually the same used in the Qlik Sense client) to help you with this.

Troubleshooting Qlik Sense extensions

Somtimes your extension does not work as expected. Here are some things you should do

1. Use your favorite browser

The browser that is built in in desktop might cache your extension files, so it is not really suited for extension development. Also in a browser you have better tools for troubleshooting and debugging. I use Google Chrome, since I think the developer tools are pretty good, but other browsers also have developer tools.

2. Look in the browser console

The console shows you if something goes wrong in your javascript code, some files are not found etc, so this is in most cases the first place to look for errors.

3. Turn off caching

When you are doing extension development the cache is your enemy: caching might mean that you are actually running an older version of your extension than you think. In Chrome you can turn it off in developer tools, in the network tab:

Note that the cache will only be disabled if you have developer tools open, so you should keep them open when developing.

4. Use the developer tools to look at your source code

Open your source code in the browsers developer tools:

Verify that this is the correct version.

5. Use the debugger to step through your code

The developer tools includes a very good debugger. Use it! Good places to set a breakpoint would be:

paint method
any methods triggered by an event (click etc)
callback methods

Note that you can also see the contents of structures like the layout and evaluate expressions used. And, of course step through your code.

6. Verify that you have only one copy of your extension under Extensions folder

Filenames of your qext file needs to be unique. If you are developing on Qlik Sense desktop it is your own responsibility to make sure that this is the case. If you make backup copies of your extension, do not save them under MyDocuments/Qlik/Sense/Extensions.

This post was originally published in Qlik Community: https://community.qlik.com/docs/DOC-8985

Qlik Sense Generic Object II: Hypercube properties

Probably the most important object you can have in your Generic Object is the HyperCube. It is what is used in most charts and is the way to tap into engine associative logic and calculation engine. Hypercubes reflect the users selections and calculations will be made on the current selections.

Hypercube properties

The hypercube properties are in a structure called qHyperCubeDef. A common setup is the one from the peoplechart extension example we have seen before:

This will create a hypercube definition with an empty array of dimensions and an empty array of measures. It will also tell engine to include 2 columns (probably 1 dimension and 1 measure) and 50 rows of data with the initial fetch of data, that is in the reply to the getLayout call. This works because the user can add dimensions and measures using the property panel, which is defined like this:

In a mashup you would in most cases need to list the dimensions and measures you want, something like this (taken from my angular-based mashup example available on GitHub:

This will create a hypercube with eight dimensions and no measures, tell engine that we want all eight columns and 400 rows in the initial fetch, set some other properties and define a call back function called setCases (not included above) that will be called every time the hypercube data has changed.

Defining dimensions

While it is possible to have a hypercube without dimensions, you probably could use a simple expression instead in most cases. So you normally would have a dimension array. This array should contain one or more NxDimenion object(found here, you need to scroll down a bit). There are two alternatives:

use a predefined dimension from the list of dimensions. If you do that you should set the qLibraryId to the id of the dimension you want to use.
use a dimension that is defined only in this dimension array. In that case you should set qDef to an object that in turn contains the dimension definition. Note that qFieldDefs is actually an array, and so can contain multiple definitions. You can actually define a cyclic group here, though I have never seen it used and don’t think it is supported in the client.

Whether you use a predefined dimension or a locally defined dimension there are some additional properties that might be useful. The most complicated is qOtherTotalSpec, which you can use to limit your hypercube to the top 10 values, or bottom 10 etc.

Adding measures

The measures array works much like the dimensions array. It also has two alternatives: referring to a predefined mease with the qLibraryId field or defining the expression in a qDef structure. Note that the measure qDef structure is not the same as the dimension qDef structure: instead of the qFieldDefs array it holds just a string field (also named qDef) with the actual expression.

Make sure you get some rows

To make sure you actually get some data in your hypercube you need to set the qInitialDataFetch property. This is actually an array, but I have never seen an example with more than one entry in the array. You should set qWidth to the number of columns (that is dimensions plus measures) in your cube and qHeight to the number of rows you need for the initial rendering of your cube.

If you do not set this property, the default is 0 rows and 0 columns, so you will get no data. A very common problem in extension development is that you add dimensions or measures and forget to update the qWidth parameter, so you do not get any values for your measures. Also rememeber that what you specify in your extension is the initial properties, so when you change your code, the objects already created will not change. A way to avoid this problem is to set qWidth to something high (10, for example) since a value that is higher than the actual number of columns will work and get you all columns.

Create a hypercube in the mashup editor

A feature not well known in Qlik Sense mashup editor is that it can help you create a hypercube. In the left column of the editor there is a button labeled ‘Add hypercube’. Select the app you want to work with and click it, and you will get a popup, something like this:

hypercubebuilder

When you click the ‘Add’ button, it will create a createCube call for you, with most of the properties filled in:

Even if you don not use the mashup editor for creating your mashup, this feature can be useful.

Qlik Sense Generic Object I: General

Getting data out of a Qlik Sense app is based on a concept called Generic Object. Behind all charts, tables and listboxes in Qlik Sense lies a generic object. Also behind many of the other things you see in the Qlik Sense client lies generic objects:

the selection toolbar
the panel on the left side of the page, with lists of dimensions, measures and master objects
the sheet is a generic object

The purpose of the Generic Object is to make it easier to make client-side changes without having to change the QIX engine. Since the Generic Object is very flexible, many client side changes do not require any changes to the engine. This is in contrast to how it works in QlikView, where all charts have different server-side implementations and client and server code are tightly coupled. This is also a reason why extensions are so powerful in Qlik Sense: to the QIX engine there is really no difference between built-in visualizations and extensions, they are all represented with a Generic Object.

Properties and layout

When you create a Generic Object you define its properties. This is a JSON structure, which you can define yourself. An example (from the peoplechart extension example):

There are two kind of properties:

user defined. Qlik only persists those and include them as-is in the layout, it is up to your code to use them for something
defined by Qlik Engine. These are validated by the Engine, and replaced in Engine output, the layout, by the calculated counterparts.

All properties defined by Engine start with a q, so it is good programming practice to let your own properties start with some other letter. The properties defined by Engine are documented here and the corresponding out data is documented here. Note that properties defined by the Qlik Sense client(like title etc) do NOT start with a q, the QIX engine does not really know what these are used for.

But the parameters known by the QIX Engine will determine the engines calculation. In the output from engine, called layout (though it’s not really the layout of a Generic Object) the definition of those properties will be replaced by the results. A hypercube definition (qHyperCubeDef) will be replaced by a actual hypercube (qHyperCube) containing the results from the calculation.

Generic Object context

Generic objects are everywhere in Qlik Sense. If you are building an extension, you define your Generic Object structure in the initialProperties property. If you are using the mashup API, you create generic objects with the createGenericObject method or one of the wrappers around it (createCube, createList, getList). In both these cases the Qlik Sense client framework will automatically refresh the Generic object with new data when it is invalidated (by for example new selections). If you are working directly on the protocol level, you need to fetch they new data yourself.

Run a reload from a Qlik Sense mashup

The main use of Qlik Sense mashup API is to visualize data from Qlik Sense apps and reuse visualizations in an app, allowing your users to interact with the data. But you can use it for other things to, like running a reload or interacting with Qlik Sense server.

But a Qlik Sense app can be used in two different contexts: desktop and server. While visualizations and interactions with those work the same in Server and Desktop, the server API (which is a REST API, unlike the QIX Engine API which works over the web socket. So the first thing we need to do is check our environment, are we in Server or Desktop. Luckily, there is a method for that, the isPersonalMode method.

Once we know if we are in Desktop or Server we can acct accordingly. If we are in Desktop, we can just call the doReload method. While this might work in Server too, we really shouldn’t use it. In a server environment you might very likely have different Qix Engines for interaction with the user and reloads. In fact it is very likely that the engine we are connected to does not have the right database connections for the reload, so running the reload there would fail. Instead we call the Qlik Server for the reload, and let it determine which computer should do the actual job. We can do this with the callRepository method.

This would look something like this:

You find a full working example in the Github repository.

This post is based on a session I originally held at Qonnections 2015, under the title ‘5 things you did not know you can do with the Mashup API’.

Get the selections in a Qlik Sense mashup

If you are using the Qlik Sense standard client there is a Selection toolbar displayed at the top of the page. It has lots of useful functionality, including search support, so in many cases you would want to use this in a mashup, which is perfectly possible.

But sometimes the selection toolbar does not fit in your web page or you want something more compact. In that case you can listen to selection changes yourself and visualize the selection state.

selections

The way to get the selection state is the getList method. An example:

The reply will contain a qSelectionObject with:

qBackCount and qForwardCount, the number of steps forward and backward that is possible
a qSelections array, with on entry for every field that is selected
the array entries will contain qField – the field name, qTotal – total number of different values in the field and qSelectedCount – the number of selected values

Using this you can make your own visualization of selection state, or perhaps hide and show visualizations based on selection states.

Access the data in a Qlik Sense visualization

When you use the getObject. call of the Qlik Sense mashup API (also known as the Capabilities API or qlik.js) it works a bit like a black box: you just call it and it will inject some Qlik content into your page , you don’t have to bother about the content, it might be any type of chart, it might even be an extension. But it you want to, you can access the data of the visualization.

The key to this is the return value from getObject. The method returns, as it says in the documentation ‘a promise of an object model’. That means you can use the return value to get access to the data in the visualization, or even call some methods to modify the visualization automatically.

Only you have to understand:

how a promise works
and how to find data in the model you eventually get

Promise

The fact that getObject returns a Promise means that you will get an object with a then method. The then method will take a parameter, a function, that is called when getObject is ready, that is after a while, normally just some milliseconds, so to you it will look like immediately, but for the computer this is a long time.

What happens when you call getObject is:

the browser will call the QIX engine over the web socket for the definition of the object with the id you supplied
when the definition comes back it will call the engine to get the data

Only after these two calls will the promise resolve, and the method you supplied as a parameter to the then method will be called. That means that at this stage the object you get will be a model with valid data.

The model

A good way to find out the contents of model is to use the debugger in your browser and inspect it. Some important parts of the model:

layout: this is the data for the visualization. Expressions, hypercubes etc will be evaluated and contain data reflecting the current selections
events, most important is the Validated event, which will be triggered when there is new valid data in the model. a full list is here ( the page refers to AngularJS-based extensions, but this is general, the same events is used for all visualizations, whether they are extensions or not)

So, to sum this up, you can get the title of the visualization like this:

You find a full working example in the Github repository.

This post is based on a session I originally held at Qonnections 2015, under the title ‘5 things you did not know you can do with the Mashup API’.

Dynamically modify a Qlik Sense visualization

One of the really powerful features of Qlik Sense API’s is the possibility to dynamically modify visualizations. You can build solutions that allows the user not only to make selections and have them reflected in all charts displayed (that’s the default, you will get that automatically if you don’t turn it off) but also to dynamically change chart properties and do things like

change dimension from for example Product to Product Lin (or to Division)
change measure from Sales Amount to Quantity or Margin or Cost
add an additional dimension and make a barchart into a stacked bar chart
modify ordering
limit the chart to only show Top 10, or bottom 5
change the chart type from Barchart to Linechart or to an extension

And all of this can be done for both built-in visualizations and extensions in the same way. The key to all of this is the applyPatches method and soft patches.

Soft patches

A soft patch is a temporary change to properties for a visualization. The fact that it is temporary means:

it will not be persisted either in Desktop or Server
it will not affect other users using the same app
once the Qix engine session is closed it will be gone, so when the user reconnects, visualizations will be back to their original state
the user can make soft patches to visualizations that me does not have the right to modify

You make sure that its a soft patch by setting the second parameter in the applyPatches call to true. The first parameter is the list of patches you want to apply. Since you might want to apply several patches, it is an array, where each entry has three values:

qPath: the path to the property you want to change, with slash (/) where javascrip/json usually has a dot
qOp: the operation you want to perform, one of “add”, “replace”, “remove”
qValue: the value you want to set. This should be a JSON value inside a string, so a boolean value would be “true” or “false”, a numeric value something like “-1” and a string (this is the tricky part) “\”new string\”” (that is a string in a string). You could also set a javascript object, use javascript standard JSON.stringify method in that case.

Perhaps this is clearer if we look at an example:

What we do in this example is that we show a list of dimensions and measures to the user. When the user clicks on one of them, we move it to be the first one in the sort order, and use applyPatches to update the array and the chart will be resorted. Since it is all done with soft patches, it will not affect other users.

You find a full working example in the Github repository.

This post is based on a session I originally held at Qonnections 2015, under the title ‘5 things you did not know you can do with the Mashup API’.