Category: protocol

Chrome developer tools in Qlik Sense extension development (part 2)

As I wrote in Part 1 Chrome Developer tools is perhaps the most important tool for a Qlik Sense extension developer. Some important points in part 1:

  • you should turn caching off in the network tab, and keep developer tools open to make sure files are not cached
  • Qlik Sense client catches program errors in your extension, so if you need to see them temporarily check the box ‘Pause on caught exceptions’. You can’t keep this box checked all the time though.

In this post we will continue with some more useful stuff.

Checking web socket traffic

As you probably know communication with the Qlik Engine is over web sockets. You find the web socket in the Network tab, just click on WS and you will find it. Click on the websocket and then on messages, and you will see the communcation, something like this:

Lots of information, lots of transactions. Some things you should know:

  • messages from client to server has a dark background, messages from server to client a white background
  • all messages are in JSON format
  • if you click on a message it will be displayed in a format that’s easier to read in the area below
  • you can filter messages by typing in text in the field where it says ‘Enter regex..’

Connect request and reply

What you see in the web socket log is simply a list of all messages sent and received, in chronological order. Since commands are processed in paralell, the reply for a specific request might not at all be anywhere near the request. Request and reply are connected with the ‘id’ attribute (that’s a feature in the json rpc protocol, which Qlik Sense is built on). This means we can filter on the id attribute to find the reply for a specific request. If you try typing in “id”:1, in the filter box you will get request #1 and it’s reply:

Handling handles

So now we know how to find a request and it’s reply, but to really understand what’s happening we also need to understand handles.

Handles is a Qlik Sense specific concept, not part of Json RPC but an addition to it, made by Qlik. The QIX protocol is object-oriented, meaning that all commands are sent to an object. When you connect to Qlik Sense engine, the only handle available is the global handle, -1.

The first thing you do in most cases is call OpenDoc to open your Qlik Sense app (or Doc, as its called in the QIX documentation). If that succeeds you will get a handle (called qHandle) back in the reply. You can them send Doc commands to this handle (almost always handle 1). If you type in the filter “handle”:1, you will see all app commands used:

As you can see there is a lot of them. Most of them fetches or creates objects in the app, with GetObject or CreateSessionObject. You do not see the replies, only the requests, since the replies do not contain the handle. But if we instead filter by the id of one of the GetObject calls, (in my case id 13) we can see what happens:

As you can see in the reply there is a new qHandle in the reply, #5. So now we can filter on “handle”:5, and see what commands were sent to this object. If you want to know what handle is used for your extension, there is as far as I know no way to get that information from standard Qlik, but you can use my Chrome Extension Add Sense.

Find commands that invalidates object

Handles are used not only to refer to objects when you send commands, but also for invalidation. When you change selections (make a new selection, clear selection, apply bookmark etc) QIX engine tells you what objects are no longer invalid. It does this with the ‘change’ array (another addition to JSON rpc) which contains handles for the invalidated objects. So if I for example want to find commands that invalidated opbject 13, I can filter on “change”:\[.*13.*\] (note: a regex, since the handle can be at any position in the array):

We only get the replies for commands that invalidated the object, to get the actual command you need to use the id field to find the actual command and handle.

Checking performance with Add Sense Chrome extension

One of the uses for the Add Sense Chrome extension is to check performance in your Qlik Sense app or mashup. To help you with this the info boxes show you a few numbers, collected as you use the extension. When you select to show the boxes, it will start collecting statistics, and you will se something like this:

Initial display with no values calculated

In this post I am using one of Qliks demo apps (you find it here, try it yourself!), so I don’t really expect to find any performance problems.

The Chrome extension will find all both objects defined in the app and session objects, but only ones that are shown on the page. For all such objects it will start collecting statistics:

  • number of recalculations made
  • latest calculation time in milliseconds
  • maximum calculation time

In the top left corner you will see the handle, that’s a number that the QIX Engine assigns to objects as they are opened. If you want to see the actual messages sent for a specific object in the web socket log you need this number. Since the handles are assigned in sequence, you will also get a grip of how many objects you are using in your solution.

Now to try this lets make some selection. Open the selection tool and click on show in the extension menu (yes the extension works in the selection tool too).Make some selections and you will see something like this (excuse me, it’s in swedish):

Selection tool with the Chrome extension enabled

Confirm your selections and close the selection tool. When you come back to the sheet probably only the app infobox will be visible. When Qlik Sense removes the objects from the page, the infoboxes disappear too. But the extension will continue to collect statistics, so click on show again in the extension menu, and you will se something like this:

Calculations have been made

Note that objects have only been recalculated once. That’s because the client only recalculate objects that are visible on the screen. The text object on the left should not be recalculated at all, since it was not affected by our selections.

So that’s what you need to do in a real situation:

  1. enable timing for objects on the screen with the show command
  2. make your selections, the extension wil register recalculations
  3. if objects have been off screen, reenable the infoboxes with the show command
  4. analyze the figures. Look for long calculation times or too many recalculations

Once you have started the timing, the only way to stop it is by refreshing the page, using the show command will not stop the monitoring.

Sharing session in Qlik Sense

Session sharing between MS Edge and Google Chrome. Note that selections made in Edge are reflected in Chrome

One of the more interesting features of Qlik Sense is the session sharing. When you open a Qlik Sense app on your iPad that you have already open on your laptop your selections are already there. And when you make a change on the iPad, the laptop window updates. It feels a bit like magic!

Of course it isn’t magic. Qlik Sense automatically shares your sessions on different devices and rather than creating a new one when you connect from the iPad, it attaches to the one you already have on your laptop. Sessions are shared between browser tabs, browsers and devices. Sharing is made based on three factors:

  • the app, of course. Sessions are app-based so the the app has to be the same for the connections to share the same sesion
  • the authenticated user. sessions are not shared between users
  • the URL used when connecting to the QIX engine

Controlling sharing

You might very well be using session sharing without really thinking about it. If you open the same app in more then one tab in the browser each tab has it’s own connection to the engine, and engine will connect them in just the same way as connections from different devices. Not that neither device or browser used affect the sharing, it’s just app, user and URL.

But sometimes you want to separate connections to have separate selection state. The way to do this is to make sure that the URL used to connect to QIX Engine is different. The standard URL used to connect to Engine has the format:

wss://[server][/proxy]/app/[app id]

  • wss for secure websocket (ws for Qlik Sense Desktop)
  • your server hostname or ip address, optionally with a port number (4848 for desktop)
  • possibly a proxy if you are on a server
  • and the app id

But you can add something more at the end, to make sure you separate your connections

An experiment

The Qlik Sense Client actually allows you to do this. You can add an identity to the end of the URL to separate sessions. Let’s make an experiment:

  1. Open an app (any app) in a browser window using the Qlik Sense client
  2. Open the same app in another browser window using a different device, a different browser or a different tab
  3. Verify that selections made in one of the windows affects the other – you have got one single session
  4. Now in one of the browser windows, modify the browser URL by adding ‘/identity/xxxx’ at the end and refresh
  5. Verify that selections made in one window does not affect the other

You have just managed to separate your connections into two different sessions! While this can be very useful to do with the standard client, for some solutions it is essential. You should really always consider this when you are building a Qlik Sense solution using the APIs.

Looking deeper

Using the browsers developer tools you can see how this is done. Open the developer tools for the browser where you added ‘/identity/xxxx’ at the end of the URL.

Click on the network tab, and filter on Web Sockets (WS). Find the connection to QIX Engine. You will see that the client has added ‘/identity/xxxx’ to the end, probably encoded to ‘%2Fidentity%2Fxxxx’.

Check the messages received on the web socket. At the begging you will find:

{“jsonrpc”:”2.0″,”method”:”OnConnected”,”params”:{“qSessionState”:”SESSION_CREATED”}}

which is a message from QIX Engine, saying that it has created a new session.

Now remove the ‘/identity/xxxx’ from the URL and refresh. Find the Web Socket again, now there is no ‘/identity/xxxx’ at the end. Check the messages again, and at the beginning you will find:

{“jsonrpc”:”2.0″,”method”:”OnConnected”,”params”:{“qSessionState”:”SESSION_ATTACHED”}}

which means that the QIX Engine found a matching session and instead of creating a new one, attached you to the existing, so you are again sharing sessions.

Masters Summit in Stockholm

Stockholm is beautiful, and April is a very good time to visit

I’m really happy to be guest speaker when Masters Summit for Qlik comes to Stockholm in April. For those of you who have not heard about Masters Summit, you should really check it out. I took part in Boston in 2017, and it was truly a great event.

API track

In Stockholm they are adding an API track, for developers working more with build applications, mashups and extension using Qlik Sense API’s and Qlik Core. Great content and led by people that really know what they are talking about. And with me in a short session on Qlik Sense use of web sockets….

Registration is here. See you in Stockholm!

Working with Qlik Sense Engine protocol: four problems you need to solve

Qlik Sense has several APIs and a .Net SDK to help you work with the QIX Engine. They help you working with Qlik Sense without having to bother about the details. But if you need to work on a lower level, there are some  problems you need to solve.

1. Connect over Web Socket

Qlik Sense QIX engine talks Web Socket, and only web socket. While you might need to make a https call to trigger authentication, the actual communication with the engine is web socket only.

The format of the URL can be seen in the Engine API Explorer, which you will find in Qlik Sense devhub:

ws://localhost:4848/app/c%3A%5Cusers%5Cerikw%5Cdocuments%5Cqlik%5Csense%5Capps%5Cexecutive%20dashboard.qvf?

Note that the app id is included in the URL, but what you get is an open web socket, not an open document. Actually, in Qlik Sense desktop it will work without the app id in the URL. That’s because the app id in the URL is not for QIX Engine. It’s for the proxy you will go through to reach Engine, and used for load balancing, the proxy uses it to find which engine to use for this app. This also means that you should always open a new web socket when you open a new app, since you need to give the proxy a chance to select the correct engine.

Conclusion:

  • you need to use web sockets to talk to QIX Engine
  • the web socket URL should contain the app id
  • you should always open a new socket for each app

2. Match request and response

With traditional HTTP communication you make a request, wait, and get a response. Web sockets work differently. You send a message to QIX Engine, but don’t wait for the answer. You just keep on sending messages, and eventually QIX Engine will reply. You do not know the order of the responses and actually the protocol does not tell you what respone belongs to which request.  Typically it looks something like this:

This is the initial communication when Qlik Sense standard client opens an app. If you think it is hard to read (and it is) you can open the browser developer tools and look in the network tab, the image above is from Chrome.

The darker rows are requests sent from the client to QIX Engine, while the white rows are responses. As you can se the client makes a series of requests (seven) and after that comes the respones. It is not obvious which responses belong to which call, and the respones can not be understood without knowing the request.

So how do you know which respons belongs to which request? Qlik Sense uses a standard called JSON-RPC for this. This means that in each request there is a field called id that contains a unique identifier for this request (in this case just a number 1, 2, 3 etc). The reply for the call contains the same id. So if you are going to build your own protocol handler you need to save the ids and use them to connect request and reply.

But note the very first line above, it has no id. Thats because there is no request for it, it is what JSON-RPC calls a notification.

Conclusion:

  • web sockets are not like traditional HTTP in that you do not wait for a response for your request
  • instead requests and responses are connected with an id included in the packages
  • you might get messages without id, in that case they are notifications

3. Handle the handles

A key concept in the QIX engine protocol is the handle. For every object you open or create you get a number, a handle. Do not confuse that with the request id (also a number) or the object id (a string, in most cases a short or long GUID). You need to save the handle when you open or create an object and use it for subsequent calls.

Initially you have only one handle, the global handle, which is -1. When you open your app you get the app handle, in most cases 1.

An example from the list above:

Conclusion:

  • QIX engine uses something called handle to keep track of objects
  • you always need a handle in your calls to QIX Engine
  • the handle will be in the response from calls that create or open an object

4. Validate when changed

Some responses from the QIX engine contains another field, ‘change’. This field is an array of numbers. The numbers are actually handles and refers to objects that have been invalidated by the call. If the user for example makes a selection, she will pretty quickly get a response. This response means that QIX engine has received the selection.

It has also determined which objects of the ones you have opened that are affected (invalidated is often the term used) by this call. This means that the data you have for these objects is no longer valid.  It is your responsibility to fetch the new data, QIX engine will not automatically recalculate it for you, only tell you that the previous version is no longer valid.

For some calls there will be no change array, while for some selections(or clear selections) a lot of open objects will be affected, so the array will be long. The apis automatically refresh the data for you, but if you are building your own stuff you need to take care of it yourself. You do that with the getLayout call for the object.

If you do not refresh your data, QIX will not tell you again that it is invalid. You only get notified once, until, of course, you get new data and that is invalidated.

Conclusion:

  • QIX engine keeps track of which open objects are affected by a call
  • the field change contains an array of these objects
  • QIX will not sendd the new data until you asks for it
  • it is your responsibility to keep track of the state of the data you have fetched from the QIX engine