Working with Qlik Sense Engine protocol: four problems you need to solve

Qlik Sense has several APIs and a .Net SDK to help you work with the QIX Engine. They help you working with Qlik Sense without having to bother about the details. But if you need to work on a lower level, there are some  problems you need to solve.

1. Connect over Web Socket

Qlik Sense QIX engine talks Web Socket, and only web socket. While you might need to make a https call to trigger authentication, the actual communication with the engine is web socket only.

The format of the URL can be seen in the Engine API Explorer, which you will find in Qlik Sense devhub:

ws://localhost:4848/app/c%3A%5Cusers%5Cerikw%5Cdocuments%5Cqlik%5Csense%5Capps%5Cexecutive%20dashboard.qvf?

Note that the app id is included in the URL, but what you get is an open web socket, not an open document. Actually, in Qlik Sense desktop it will work without the app id in the URL. That’s because the app id in the URL is not for QIX Engine. It’s for the proxy you will go through to reach Engine, and used for load balancing, the proxy uses it to find which engine to use for this app. This also means that you should always open a new web socket when you open a new app, since you need to give the proxy a chance to select the correct engine.

Conclusion:

• you need to use web sockets to talk to QIX Engine
• the web socket URL should contain the app id
• you should always open a new socket for each app

2. Match request and response

With traditional HTTP communication you make a request, wait, and get a response. Web sockets work differently. You send a message to QIX Engine, but don’t wait for the answer. You just keep on sending messages, and eventually QIX Engine will reply. You do not know the order of the responses and actually the protocol does not tell you what respone belongs to which request.  Typically it looks something like this:

This is the initial communication when Qlik Sense standard client opens an app. If you think it is hard to read (and it is) you can open the browser developer tools and look in the network tab, the image above is from Chrome.

The darker rows are requests sent from the client to QIX Engine, while the white rows are responses. As you can se the client makes a series of requests (seven) and after that comes the respones. It is not obvious which responses belong to which call, and the respones can not be understood without knowing the request.

So how do you know which respons belongs to which request? Qlik Sense uses a standard called JSON-RPC for this. This means that in each request there is a field called id that contains a unique identifier for this request (in this case just a number 1, 2, 3 etc). The reply for the call contains the same id. So if you are going to build your own protocol handler you need to save the ids and use them to connect request and reply.

But note the very first line above, it has no id. Thats because there is no request for it, it is what JSON-RPC calls a notification.

Conclusion:

• web sockets are not like traditional HTTP in that you do not wait for a response for your request
• instead requests and responses are connected with an id included in the packages
• you might get messages without id, in that case they are notifications

3. Handle the handles

A key concept in the QIX engine protocol is the handle. For every object you open or create you get a number, a handle. Do not confuse that with the request id (also a number) or the object id (a string, in most cases a short or long GUID). You need to save the handle when you open or create an object and use it for subsequent calls.

Initially you have only one handle, the global handle, which is -1. When you open your app you get the app handle, in most cases 1.

An example from the list above:

Conclusion:

• QIX engine uses something called handle to keep track of objects
• you always need a handle in your calls to QIX Engine
• the handle will be in the response from calls that create or open an object

4. Validate when changed

Some responses from the QIX engine contains another field, ‘change’. This field is an array of numbers. The numbers are actually handles and refers to objects that have been invalidated by the call. If the user for example makes a selection, she will pretty quickly get a response. This response means that QIX engine has received the selection.

It has also determined which objects of the ones you have opened that are affected (invalidated is often the term used) by this call. This means that the data you have for these objects is no longer valid.  It is your responsibility to fetch the new data, QIX engine will not automatically recalculate it for you, only tell you that the previous version is no longer valid.

For some calls there will be no change array, while for some selections(or clear selections) a lot of open objects will be affected, so the array will be long. The apis automatically refresh the data for you, but if you are building your own stuff you need to take care of it yourself. You do that with the getLayout call for the object.

If you do not refresh your data, QIX will not tell you again that it is invalid. You only get notified once, until, of course, you get new data and that is invalidated.

Conclusion:

• QIX engine keeps track of which open objects are affected by a call
• the field change contains an array of these objects
• QIX will not sendd the new data until you asks for it
• it is your responsibility to keep track of the state of the data you have fetched from the QIX engine