api – Extending Qlik

Trying Qlik Sense Business API’s

Much of Qlik’s focus is today on their cloud products. Getting up and running with Qlik Sense Business (the name Qlik uses for their entry-level cloud product) is easy. And even if the hub is different and there is no QMC, it feels familiar. After all it’s the same engine behind the scenes. Many (probably most) extension will also work, since the extension API is the same, and the Capabilities API also the same – almost. But even though the methods are all there, some of them actually don’t give the same result.

The global object in the Capabilities API

The differences I have found are in the global object, as should be expected. (documentation is here). So let’s try it.

The first method is getAppList(). In Qlik Sense Desktop or Qlik Sense Enterprise (which is Qlik’s name on the product you install yourself on a windows server) this method will give you a list of all apps the currently authorized user has access to. In Qlik Sense Business it still works, but if you try it, it looks something like this:

In my Qlik Sense Business hub, I’ve got four apps, but getAppList only returns the one I am currently using. So if your extension uses getAppList to create a list of other apps, it won’t work in Qlik Sense Business.

Another frequently used method in the global object is getAuthenticatedUser(). You use it to find out the users id. In Qlik Sense Desktop you’ll get ‘Personal\\Me’, in Qlik Sense Enterprise you’ll get a string containing UserId and UserDirectory. In qlik Sense Business it looks like this:

So, while the format is the same as in Qlik Sense Enterprise, the content is different. The UserDirectory is empty, and the UserId is actually the IdP subject you can find in the cloud console (IdP stands for Identity Provider).

callRepository

The Capabilities APIs has one method designed to call the repository, originally built to call Qlik Sense Enterprise REST API called callRepository(). It is marked as Deprecated, but it is still there in Qlik Sense Enterprise, and looks like it’s there in Qlik Sense Business too. What happens if we try it?

Qlik Sense Cloud has the REST endpoint /api/v1/items to list apps the user has access to. If we try it, it looks like this:

We do get a warning in the console, but the call works, the data array actually contains the four apps I’ve got in my Qlik Sense Business hub. But since it is deprecated we probably shouldn’t use it but use some standard method of making a REST call instead.

AddSense Chrome 1.2.1: active objects, dimensions, measures, link to Catwalk

A new version of my Chrome extension for Qlik Sense, Add Sense, is now available in Chrome Web Store. This is a pretty big version, actually that’s why it has number 1.2.1, because a bug sneaked into 1.2.0 which I had to fix.

New app level operations

You reach the new functions from the app popup box:

In fact, most of the buttons are new. From the left to the right they are used for:

showing the script
listing the tables
exploring the data model (in Catwalk)
listing variables
listing dimensions
listing measures
listing extensions used
listing active objects

Data model

A number of the new functions helps you with the data model. You can see what tables are defined in the app, and list dimensions and measures (and copy their definition into a program of your choice, like Excel or something). But perhaps the best new function is the link to Catwalk. Click on the compass icon, and it will open Catwalk with your current app:

If you don’t know what Catwalk is it’s time to get to know it knopw. It’s an open source tool from Qlik that helps you explore the data model of a Qlik Sense app. If it doesn’t work on your server, you might need to whitelist it. Check the documentation! Originally I planned to do some visualization of the data model, but I realized that I cannot make anything better than Catwalk, so why not reuse it.

List all active objects

The Add Sense extension uses Angular and some CSS selectors to find Qlik objects in your page, whether it’s the standard Qlik Sense client or a mashup. This only works for visualizations that are either Qlik Sense standard visualizations or extensions built with the traditional Angular-based method. But in a custom built web app using Qlik you might choose to visualize the data in some other way. Qlik Sense objects used this way have previously not been available in Add Sense, you could not check their properties, monitor recalculation times etc.

With the new function to list all active objects that has changed. While the little boxes showed on the page start from the HTML DOM and Angular scopes, this function uses enigmajs and lists all objects active in the enigmajs session. It might look something like this:

Example from the demo app ‘History of the fortune 500’

It still needs to find the enigmajs session, which might not work in all cases. If you want to use the extension for a case where it does not work, please let me know, you might be able to fix that with just a very small change.

Dimensions and measures

The extension can now also list dimensions and measures.

You can use the copy function in the dialog to copy the contants to the clipboard and then into application of your choice, like Excel etc.

And that’s it for the new functions. Please let me know if there is anything you are missing.

Who am I? Getting the user id in a Qlik Sense Extension or mashup

Sometimes you need to know who the user is in your Qlik Sense extension or mashup. Perhaps you want to help the user filling in some data, display notifications to the user or something else that I can’t even think of.

getAuthenticatedUser call

There is no property containing user id (at least not someone that I know of), you’ll need to call an API one way or the other. The relevant call in the API is getAuthenticatedUser(), available on the global object.

In an extension, you should use the global property available on the app, so the call would be app.global.getAuthenticatedUser(). In a mashup it depends.. If you need to do this before the app is opened, you could use qlik.getGlobal(config) to get the global object. But be aware that this will mean another open web socket. If you can wait, it’s better to use app.global, since that means you will reuse the existing web socket.

Like most API calls, getAuthenticatedUser() is asynchronous, so you need to call it like this:

qlik.currApp(this).global.getAuthenticatedUser().then(function(reply){
    console.log('user',reply.qReturn);
});

If you run this in Qlik Sense Desktop, you should get the string ‘Personal\Me’. But in Qlik Sense Enterprise it’s a bit more complicated. Ehen I run it I get: ‘UserDirectory=UPPER88QS; UserId=erik’. So this is a string, which contains two key-value pairs. You’ll need to parse this, to grab the UserId, or if you really want both values (perhaps you have more than one UserDirectory? Perhaps this is different between production – test – development?

Using OSUser()

But there is another method. Youd could use the Qlik Script system function OSUser(). In a mashup this is probably not a great idea, since you would need to either create a generic object containing OSUser(), or call the Evaluate method, not even included in Capabilities API, only in Enigma.

But in an extension the framework always creates a generic object for you and gets the layout. So you could add ‘OSUser’ to your initialProperties, and get the result in your layout, something like this:

initialProperties: {
   user: {
      qStringExpression: '=OSUser()'
   }
},

In most cases you would of course have other stuff in your properties too, and they can easily be combined. Don’t show this in your property panel, if you don’t want your users to be able to change it.

This will mean that you in layout.user will have the user the same way as when you called getAuthenticatedUser(), you will probably need to parse it, but you don’t have to handle the fact that is it asynchronous.

A warning

This is all client side. That means it’s not really safe, you should not use this to restrict user access – it’s OK to use this to hide operations from the user that they should not be allowed to use, but you need to verify the user serverside too, in a safe way, that can not be hacked. But that’s another topic…

Qlik Sense Global object

Sometimes when you are working with the Qlik Sense APIs you need access to information that is not connected to the Qlik Sense document you are working with, but rather to the QIX Engine running.

You find the apis for this in the global object in the Capabilities APIs, or in the Global class in the Engine API. While the Capabilities APIs only include very few methods, the full list of calls in the Engine API is much longer.

Finding the Global object

While there is a method in the Capabilities API to get the global object, in most cases you should not use it. In fact if you are building an extension you never should. Instead you should use the global property on the app.

That is because the getGlobal() method will open a new web socket to the engine, while app.global property will use the web socket used for the app. There is not reason to open another web socket to the server when you already have one, and in an extension there always is one.

The only situation getGlobal() should be used is when you do not have an open web socket, like in a mashup before the app is opened. And in that case you might be better off with a REST call.

Chrome developer tools in Qlik Sense extension development (part 2)

As I wrote in Part 1 Chrome Developer tools is perhaps the most important tool for a Qlik Sense extension developer. Some important points in part 1:

you should turn caching off in the network tab, and keep developer tools open to make sure files are not cached
Qlik Sense client catches program errors in your extension, so if you need to see them temporarily check the box ‘Pause on caught exceptions’. You can’t keep this box checked all the time though.

In this post we will continue with some more useful stuff.

Checking web socket traffic

As you probably know communication with the Qlik Engine is over web sockets. You find the web socket in the Network tab, just click on WS and you will find it. Click on the websocket and then on messages, and you will see the communcation, something like this:

Lots of information, lots of transactions. Some things you should know:

messages from client to server has a dark background, messages from server to client a white background
all messages are in JSON format
if you click on a message it will be displayed in a format that’s easier to read in the area below
you can filter messages by typing in text in the field where it says ‘Enter regex..’

Connect request and reply

What you see in the web socket log is simply a list of all messages sent and received, in chronological order. Since commands are processed in paralell, the reply for a specific request might not at all be anywhere near the request. Request and reply are connected with the ‘id’ attribute (that’s a feature in the json rpc protocol, which Qlik Sense is built on). This means we can filter on the id attribute to find the reply for a specific request. If you try typing in “id”:1, in the filter box you will get request #1 and it’s reply:

Handling handles

So now we know how to find a request and it’s reply, but to really understand what’s happening we also need to understand handles.

Handles is a Qlik Sense specific concept, not part of Json RPC but an addition to it, made by Qlik. The QIX protocol is object-oriented, meaning that all commands are sent to an object. When you connect to Qlik Sense engine, the only handle available is the global handle, -1.

The first thing you do in most cases is call OpenDoc to open your Qlik Sense app (or Doc, as its called in the QIX documentation). If that succeeds you will get a handle (called qHandle) back in the reply. You can them send Doc commands to this handle (almost always handle 1). If you type in the filter “handle”:1, you will see all app commands used:

As you can see there is a lot of them. Most of them fetches or creates objects in the app, with GetObject or CreateSessionObject. You do not see the replies, only the requests, since the replies do not contain the handle. But if we instead filter by the id of one of the GetObject calls, (in my case id 13) we can see what happens:

As you can see in the reply there is a new qHandle in the reply, #5. So now we can filter on “handle”:5, and see what commands were sent to this object. If you want to know what handle is used for your extension, there is as far as I know no way to get that information from standard Qlik, but you can use my Chrome Extension Add Sense.

Find commands that invalidates object

Handles are used not only to refer to objects when you send commands, but also for invalidation. When you change selections (make a new selection, clear selection, apply bookmark etc) QIX engine tells you what objects are no longer invalid. It does this with the ‘change’ array (another addition to JSON rpc) which contains handles for the invalidated objects. So if I for example want to find commands that invalidated opbject 13, I can filter on “change”:\[.*13.*\] (note: a regex, since the handle can be at any position in the array):

We only get the replies for commands that invalidated the object, to get the actual command you need to use the id field to find the actual command and handle.

Guide to Qlik Sense ApplyPatches

One of the more powerful calls in the QIX Api is the ApplyPatches call. It allows you to dynamically add, change or remove basically any property of a generic object. With it, you can allow users to dynamically change visualizations: reorder data, change dimensions and measures, add limitation etc etc. But using it in your implementation can be tricky.

Parameters

The ApplyPatches call only has two parameters (plus the handle, identifying which object we are patching:

Name	Description	Mandatory	Type
qPatches	Array of patches.	Yes	Array of NxPatch
qSoftPatch	Patches are not saved	No	Boolean

So it’s an arry and a boolean (true/false) flag. Let’s start with the flag. Setting the qSoftPatch flag to true does mean that the patched property is not saved, but it also has some other consequences:

you can change objects that you otherwise cannot, like published objects, or objects in apps you cannot change
the changes will only affect you, other users will not see them, they might very well apply a completely different patch to the same object
the changes will only affect the current session. If you disconnect and connect again, they will be lost and you will start over with the original version of the object

So if your goal is to allow users to dynamically change visualizations you would want qSoftPatch to be true. While the default value false works OK in Qlik Sense Desktop, it’s generally not something you would want to use in a server environment.

The array of patches

The array is a bit more difficult. It’s an array so you can change several properties with one call. For example if you change dimension or measure you should consider changing the title too, and probably do that in the same call. Every entry(almost) in the array has three parameters:

Name	Description	Type
qOp	Operation to perform. Add, remove or replace.	String
qPath	Path to the property.	String
qValue	The value of the property	String

The first parameter, qOp, is an easy one. In most cases you want to change a property, use ‘replace’. Occasionaly you want to add a property, use ‘add’. Rarely you want to remove a property, in that case use ‘remove’.

The second parameter, qPath, is the path to the property, using / between the property names (not . as in javascript). Remember:

you’re patching properties, not the layout, so it’s /qHyperCubeDef/… not qHyperCube
for arrays you need to include the position in the array, and numbering starts with zero, so first dimension is /qHyperCubeDef/qDimensions/0/ etc
property structures can be complicated, speciallly the hypercube..
you can use my Chrome extension Add Sense to create the path argument. Show properties for the objects, switch to patch mode, copy the path

The second parameter, qPath, is the path to the property, using / between the property names (not . as in javascript). Remember:

you’re patching properties, not the layout, so it’s /qHyperCubeDef/… not qHyperCube
for arrays you need to include the position in the array, and numbering starts with zero, so first dimension is /qHyperCubeDef/qDimensions/0/ etc
property structures can be complicated, speciallly the hypercube..
you can use my Chrome extension Add Sense to create the path argument. Show properties for the objects, switch to patch mode

Some examples

A common case is when you want your users to be able to change the ordering of for example a table dynamically. Ordering of a hypercube is determined by the qInterColumnSortOrder property, an array with the column numbers in the order they should be used for sorting. If you want say the third column (which has number 2, since numbering starts with 0) you move the number 2 to the beginnging of the array and call applyPatches:

model.applyPatches( [{
  qPath: '/qHyperCubeDef/qInterColumnSortOrder',
  qOp: 'replace',
  qValue: JSON.stringify(sortorder)
}], true );

You can find a more complete example in the mashup I showed at Qonnections back in 2015.

Another example is when you want to reverse the ordering of a hypercube. You do this by toggling the boolean value of qReverseSort in the dimension or measure where you want to change the order. For the first dimension this will be:

model.applyPatches( [{
  qPath: '/qHyperCubeDef/qDimensions/0/qDef/qReverseSort',
  qOp: 'replace',
  qValue: JSON.stringify( !reversesort )
}], true );

Qlik Sense developer tool for Chrome now available

The Chrome extension activated in the built-in Qlik Sense client

Some years ago I developed a Qlik Sense developer tool extension, mainly to help mashup developers with the problem of finding object id’s to use in their mashup. Over time we added more features to it, so you could now access script and variables, and use it as a tool (of several) for performance troubleshooting.

But it always had the problem that you needed to add the extension to the Qlik Sense app, something that is not always possible or practical. What if we could do this without needing to add anything to the app?

That’s the reasoning behind the Qlik Sense developer tool now available in the Chrome web store. It’s still an extension, but now a Chrome extension and not a Qlik Sense extension. That means if you install it in your browser, it will always be available.

Another advantage with this approach is that it also works with Qlik Sense mashups, using Qlik Sense Capabilities API.

The Chrome extension activated in a mashup

Still you can find object id’s with the extension, and explore properties for objects, sheets and apps. You can also read the script (provided you have access rights) and the variables. And you can now see what extensions and charts are actually used in an app, something that is not easy with the standard client.

You find more information here.

Sharing session in Qlik Sense

Session sharing between MS Edge and Google Chrome. Note that selections made in Edge are reflected in Chrome

One of the more interesting features of Qlik Sense is the session sharing. When you open a Qlik Sense app on your iPad that you have already open on your laptop your selections are already there. And when you make a change on the iPad, the laptop window updates. It feels a bit like magic!

Of course it isn’t magic. Qlik Sense automatically shares your sessions on different devices and rather than creating a new one when you connect from the iPad, it attaches to the one you already have on your laptop. Sessions are shared between browser tabs, browsers and devices. Sharing is made based on three factors:

the app, of course. Sessions are app-based so the the app has to be the same for the connections to share the same sesion
the authenticated user. sessions are not shared between users
the URL used when connecting to the QIX engine

Controlling sharing

You might very well be using session sharing without really thinking about it. If you open the same app in more then one tab in the browser each tab has it’s own connection to the engine, and engine will connect them in just the same way as connections from different devices. Not that neither device or browser used affect the sharing, it’s just app, user and URL.

But sometimes you want to separate connections to have separate selection state. The way to do this is to make sure that the URL used to connect to QIX Engine is different. The standard URL used to connect to Engine has the format:

wss://[server][/proxy]/app/[app id]

wss for secure websocket (ws for Qlik Sense Desktop)
your server hostname or ip address, optionally with a port number (4848 for desktop)
possibly a proxy if you are on a server
and the app id

But you can add something more at the end, to make sure you separate your connections

An experiment

The Qlik Sense Client actually allows you to do this. You can add an identity to the end of the URL to separate sessions. Let’s make an experiment:

Open an app (any app) in a browser window using the Qlik Sense client
Open the same app in another browser window using a different device, a different browser or a different tab
Verify that selections made in one of the windows affects the other – you have got one single session
Now in one of the browser windows, modify the browser URL by adding ‘/identity/xxxx’ at the end and refresh
Verify that selections made in one window does not affect the other

You have just managed to separate your connections into two different sessions! While this can be very useful to do with the standard client, for some solutions it is essential. You should really always consider this when you are building a Qlik Sense solution using the APIs.

Looking deeper

Using the browsers developer tools you can see how this is done. Open the developer tools for the browser where you added ‘/identity/xxxx’ at the end of the URL.

Click on the network tab, and filter on Web Sockets (WS). Find the connection to QIX Engine. You will see that the client has added ‘/identity/xxxx’ to the end, probably encoded to ‘%2Fidentity%2Fxxxx’.

Check the messages received on the web socket. At the begging you will find:

{“jsonrpc”:”2.0″,”method”:”OnConnected”,”params”:{“qSessionState”:”SESSION_CREATED”}}

which is a message from QIX Engine, saying that it has created a new session.

Now remove the ‘/identity/xxxx’ from the URL and refresh. Find the Web Socket again, now there is no ‘/identity/xxxx’ at the end. Check the messages again, and at the beginning you will find:

{“jsonrpc”:”2.0″,”method”:”OnConnected”,”params”:{“qSessionState”:”SESSION_ATTACHED”}}

which means that the QIX Engine found a matching session and instead of creating a new one, attached you to the existing, so you are again sharing sessions.

My best advice to Qlik Sense extension developers

When you work a lot with something you develop opinions about how things should be done. Here is my list, after two years as an independent contractor, mainly with Qlik Sense extensions and mashup, and of course based on my experience from Qlik.

Git is the industry standard for version control, and it’s free.

1. Use version handling

There is no reason not to use version handling of your code. I use Git for everything, but if you are still using something else, that might be OK too. You probably should consider switching to Git, but as a developer that might be a decision taken over your head.

The benefits of using version control are enourmous. You can easily go back and find the reason behind a bug or the implementation behind that feature the customer now wants in another extension too. Invaluable!

You also should keep a version number in your qext file, and change it every time you deploy a new version. The main benefit of this is that you can see what version of your extension is in use in a Qlik Sense installation. I have described how to do this in a previous post.

2. Always test your extension in a browser

The developer console is an excellent tool for extension development. You use it to debug your javascript code, to inspect the HTML and CSS and to analyze the network traffic, including the web socket traffic. It can also be used to turn caching off, so you know that you are running the latest version. The browser included with Qlik Sense Desktop is not of much use for extension development. Even if you can open the console, it’s an environment your users do not use.

3. Stay in your sandbox

The extension framework gives you a HTML element for your extension. Stay inside it, unless you have very good reasons not to do so. That goes both for your javascript and CSS.

do not inject HTML elements outside of your own element
prefix your CSS rules so that it affects only the extension
scan only your element, use $element.find() rather than $(), or element.querySelectorAll rather than document.querySelectorAll
avoid html id’s, since they must be unique within the page. If you must use one (really only when you use a library that needs an id) make sure they are unique for each extension instance (test with multiple instances of the same extension)

There are occasions when you need to break theses rules, but only do it when it’s need for your extension functions.

4. Use modern development tools

Pretty soon after you have started with extension development you will grow out of the Qlik Sense dev-hub extension editor. A modern, nodejs-based environment, with a good text editor gives you a lot of benefits, and works well with version handling software. I would recommend the following:

a good editor. I mostly use Visual Studio Code, but there are others that probably are just as good.
a CSS preprocessor like less helps you with prefixing css, add vendor prefixes etc
a lint tool, to find common errors an enforce good programming practices
possibly a preprocessor, to allow you to use modern javascript features and still work on older browsers

5. Always keep your extension backwards compatible

After a while your users will most likely want more functionality in your extension. They might want more interactivity, more rendering options, perhaps support for more dimensions and measures. You will probably add new settings giving your extension more flexibility. You might also find that some of the stuff you originalyy programmed was not perfect (that happens to all of us..). But if you have deployed your extension to production, make sure that all your changes are backward compatible. If you don’t you will have huge problems when you deploy the new version, with breaking apps.

If you find it impossible to make your new extension backwards compatible you probably should not be making a new version, but a new extension, with a different name.

6. Base your rendering on only the layout

The model behind Qlik Sense extensions (and the built-in charts too, actually) is that the extension is based on a Generic Object, where you configure the Generic Object in the property panel and use the layout for rendering. Stick to that model, do not break it. If you need more data for your rendering, add it to the properties structure. Avoid:

creating HyperCubes and ListBox objects programatically, add them to initialProperties and the property panel instead
fetching variable values programatically, add expressions to the property structure instead (that also gives the app developers more options, like hard-coding values, or making them change automatically as selection state (or data) changes

It is however OK, even recommended, to call API methods for:

showing lists of available values in the property panel
making selections etc when the users clicks on buttons, menues or other parts of your UI

I have written on this before here and here.

A third way to run a reload from a Qlik Sense mashup

Back in 2016 I wrote a post on running a reload from a Qlik Sense mashup. To recap, I described two ways:

call the Engine API method doReload(). Use this in Qlik Sense desktop, since it’s the only method available.
use the repository call App reload optionally with the help of the callRepository method. This is of course only relevant in a server installation.

You’ll find code examples in my original post, and also how to know if you are running in server or desktop.

What happens when you use /app/reload ?

While this method works well in some scenarios, like when you just want to run one reload of an app ASAP, it has some limitations. Behind the scenes Qlik will create a very simple task for you, and try to start it immediately. The task will look like this:

Very basic, just an app, a name, an the default Task session timeout of a day (or 1440 minutes). No triggers, since the task is started by the REST call, no tags, no custom properties. And, perhaps most important, no retries. This means that if you create a series of reloads this way, most likely only the first few will succeed, the rest will fail since there is no engine instance available.

Another method to start a reload

But there is another way to do it. You can easily create the Task yourself, and set whatever parameters you need. Qlik Sense QRS API is really easy to work with, and the callRepository method can help you by fixing the xrfkey and wrapping the call in a javascript Promise. So creating a reload task can be made as simply as this:

This will create the reload task the way we want it and return the id for the task, whish we can use to start it. That’s just another call to the Repository like this:

And that’s it. A little more code than the second method, but much more control over what we get.