Work status

Things have changed! In December of 2022 I finished the pretty large project which has taken up most of my time for the last two years. This means that I can now make room for other projects, even if full-tme projects are still not possible.

I also hope to have some time for this blog, and my Chrome extensions. And there are some other pretty cool stuff on the way.

Remote contracts are the easiest to find time for, and most of the work I do can be made remotely, so please get in touch if you need help with Qlik Sense mashups, extensions or integration. Stuff I can help you with:

  • development of mashups and extensions
  • lower level integration using the Qlik Engine API, possibly together with other APIs
  • extension and mashup strategy: when and how to use them, what to check for in an extension or mashup
  • maintenance of extensions
  • troubleshooting and performance problems
  • validating (and fixing) extensions and mashups with new Qlik Sense versions

Recently I have helped customers and partners with among other things:

  • make extensions work with NPrinting (pretty common problem, I’m afraid)
  • building a generic write-back solution with extension and back-end service
  • advanced extension development
  • making sure extensions work on a mashup hosted on a web server (not Qlik)
  • trobleshooting performance problems in a mashup, using my Chrome add-on Add Sense
  • moving solutions to Qlik Sense SaaS environment

Best way to reach me is through mail: erik at upper88.com

Who am I? Getting the user id in a Qlik Sense Extension or mashup

Sometimes you need to know who the user is in your Qlik Sense extension or mashup. Perhaps you want to help the user filling in some data, display notifications to the user or something else that I can’t even think of.

getAuthenticatedUser call

There is no property containing user id (at least not someone that I know of), you’ll need to call an API one way or the other. The relevant call in the API is getAuthenticatedUser(), available on the global object.

In an extension, you should use the global property available on the app, so the call would be app.global.getAuthenticatedUser(). In a mashup it depends.. If you need to do this before the app is opened, you could use qlik.getGlobal(config) to get the global object. But be aware that this will mean another open web socket. If you can wait, it’s better to use app.global, since that means you will reuse the existing web socket.

Like most API calls, getAuthenticatedUser() is asynchronous, so you need to call it like this:

qlik.currApp(this).global.getAuthenticatedUser().then(function(reply){
    console.log('user',reply.qReturn);
});

If you run this in Qlik Sense Desktop, you should get the string ‘Personal\Me’. But in Qlik Sense Enterprise it’s a bit more complicated. Ehen I run it I get: ‘UserDirectory=UPPER88QS; UserId=erik’. So this is a string, which contains two key-value pairs. You’ll need to parse this, to grab the UserId, or if you really want both values (perhaps you have more than one UserDirectory? Perhaps this is different between production – test – development?

Using OSUser()

But there is another method. Youd could use the Qlik Script system function OSUser(). In a mashup this is probably not a great idea, since you would need to either create a generic object containing OSUser(), or call the Evaluate method, not even included in Capabilities API, only in Enigma.

But in an extension the framework always creates a generic object for you and gets the layout. So you could add ‘OSUser’ to your initialProperties, and get the result in your layout, something like this:

initialProperties: {
   user: {
      qStringExpression: '=OSUser()'
   }
},

In most cases you would of course have other stuff in your properties too, and they can easily be combined. Don’t show this in your property panel, if you don’t want your users to be able to change it.

This will mean that you in layout.user will have the user the same way as when you called getAuthenticatedUser(), you will probably need to parse it, but you don’t have to handle the fact that is it asynchronous.

A warning

This is all client side. That means it’s not really safe, you should not use this to restrict user access – it’s OK to use this to hide operations from the user that they should not be allowed to use, but you need to verify the user serverside too, in a safe way, that can not be hacked. But that’s another topic…

Pros and cons of Qlik Sense mashups

Recently I got the question on if I would recommend using Qlik Sense mashups. That got me thinking, of course the question is not so simple: it all depends. As with most things there are pros and cons…

Why would you use Qlik Sense mashups?

There are several reasons for using Qlik Sense mashups. A very common one is that you want to apply your companys profile and have a look-and-feel more like that you have for other systems. When I started building mashups for customers Qlik Sense hade no support for theming, so if you wanted to change things like fonts you needed to do that in a mashup. Now you can do this in a theme, but I honestly don’t know how much custom themes are used today.

Another common reason is that you want navigation to work another way than it does in Qlik Sense. Perhaps put sheets in a hierarchical structure or have views targeted to different user categories.

You might also mobile support to work differently than it does in standard Qlik Sense. In a mashup you can leverage open source CSS framework like Bootstrap or Bulma which gives you possibilities you do not have in the built-in Qlik Sense client.

Another reason is to simplify the user interface. In a mashup you can focus on the workflow that’s most important for your users (usually analyse data) and skip features like edit mode, storytelling, data load and modelling. This could potentially let your users get started faster and reduce the need for training, a big cost in a large dashboard projects.

You might also want to add features not in the standard Qlik Sense client. Examples could be commenting, writeback, sharing, exports not covered in the standard client, integration with other systems. Features like alternate state and default (startup) bookmark used to fall in this category, but are now included in the standard clent.

Problems with using Qlik Sense mashups

You should realize that mashups require HTML, CSS and javascript knowledge, a skillset most Qlik developers don’t have. You need to make sure you have both web developer skills and Qlik skills in your organization, whether you have that inhouse or work with a partner. And even though there are lots of web developers out there, very few of them know how to work with the Qlik APIs.

And the javascript environment is changing fast. While Qlik script has changed little in the 12 years I have worked with Qlik, the javascript environment has changed a lot. The language has a lot of new features, and frameworks, build tools and libraries has changed even more, so that the environment used in many mashups, with angularjs 1.x, today is pretty old and completely different frameworks (React etc) dominate, at least in new projects.

The tools included with Qlik Sense, like dev-hub, the Mashup Editor and the templates included makes it easy to create a mashup with only drag-and-drop and really no programming, but for production level projects it really is not enough. You would want stuff like multi-page support (including only loading Qlik Sense objects when they are needed), version handling and a dev environment where you can use npm packages, build tools etc.

Some advice on mashup development

So, you have decided that your use case is best solved with mashups. But how should you go about developing it? Here are some advice based on my experience with building mashups.

Recognize that you need web developer skills. The mashups itself is largely a web developer project. You need experience on web development, and on troubleshooting and debugging web apps. Someone with deep knowledge of HTML, CSS, Javascript and Qlik APIs is needed.

Web developer skills is not all. For users the most important part of your mashup is the data. For that you still need Qlik developer skills. In many cases you would have two kind of developers, those building charts and those working on the mashup itself.

Use version control. I used to say that version control is what differentiate professional development from private hacks. Today this is no longer true, even many hobby projects use version control. Set it up early in your project, commit changes often, make suire you know exactly what code runs in production.

Separate Qlik Object definitions from the javascript code. Try to avoid having Qlik Sense object definitions, field names etc in the actual code. Separate object id’s etc into separate configuration files or use tags in the Qlik Sense apps. This improves reusability of your mashup code and simplifies things a lot.

Use a CSS framework. A CSS framework like Bootstrap or Bulma helps you a lot with making your mashup responsive and good-looking. You might want to modify it by overriding the styles with your profile colors, fonts etc, but add those later.

Files in Qlik Sense extensions

Qlik Sense allows you to add extensions to your installation. In a server environment you do this by importing a zip file, containing one or several extensions. In Qlik Sense Desktop you can unzip (the same) zip file to a directory under Documents/Qlik/Sense/Extensions.

The qext file

The qext file is what defines the extension. It’s a JSON file, it has to be valid JSON, but very few properties are mandatory. The qext file defines the extension root, Qlik Sense will make files the directory where the qext file and all directories under it available for HTTP calls to the server. So if you add an extension with the file xxxx.qext, you can fetch the file from the server with:

HTTP GET /extensions/xxxx/xxxx.qext

And all other files in the extension can also be fetched the same way:

HTTP GET /extensions/xxxx/xxxx.js

Sub-directories can also be used:

HTTP GET /extensions/xxxx/lib/jquery.js

Not that the directory name on disk is not relevant. On Qlik Sense Desktop you can build your own directory structure. Extensions in Dashbord bundle and Visualization bundle are kept in their own sub-directory, which doesn’t affect the URL used to load them. But if you do build a structure for your extensions, be careful that you do not keep duplicates of the same extensions. If you do you will not know which copy is actually loaded.

Multiple qext files

You can also have multiple qext files in the same zip file, which will mean that several extensions, and their files, are made available. Qlik Sense Enterprise will not allow you to have more than one qext file in the same directory, you can do this in Qlik Sense Desktop, but it is confusing, so you probably shouldn’t.

Multiple qext files is useful when you want to use the same files in more than one extension, like some library, CSS files etc. You can then put common files in one directory and add a qext file in it to make it available for all extensions. By putting them in the same zip file you make sure that they are updated at the same time as the extensions using the library. Another effect is that if the user is using two different extensions that use the same library, files will only be downloaded once. If every extension has it’s own copy of a library, it will be downloaded multiple times.

For Qlik Sense Enterprise you have no control over how the extension files are stored. It is not hard to find the files on disk, but the repository is responsible for storing them and serving to the client.

The extension type

One of the fields you always should have in the qext file is the type. In a visualization extension, that you should be able to use in the Qlik Sense client, it should be set to ‘visualization’:

"type": "visualization",

If you set it to something else, the Qlik Sense client will not load it and your users will not be able to create visualizations using it.

For mashups it is usually set to ‘mashup’:

"type": "mashup",    

This is not absolutely necessary, your mashup will still work even if you set the type to something else, but if you want to manage it and possibly edit the file in Qlik Sense dev-hub you need to set the type to ‘mashup’. That also means that if you want to hide your mashup in dev-hub, set the type to something else (perhaps ‘webapp’ ?).

There are also a few other types used by Qlik Sense: ‘visualization-template’ and ‘mashup-template’. These types are used for the templates used in dev-hub when you create a new visualization or mashup.

If you plan to build several mashups creating a good mashup template with the look-and-feel you want might be a good choice.

The wbfolder.wbl file

One of the most misunderstood file in extensions is the wbfolder.wbl file. It is simply a list of files in the extension, with one line for every file, terminated by semicolon:

com-qliktech-horizlist.js;
com-qliktech-horizlist.qext;
horizlist.css;

The wbfolder.wbl file is not mandatory. It is not needed for your extension for work, if you do have one it does not need to contain all files. It is only used by the dev-hub editors. Without it you cannot open or duplicate the extension in dev-hub.

If you do have it, only the files listed in wbfolder.wbl will be available in the editors. Also only the files in wbfolder.wbl will be copied if you make a copy of the extension or mashup. The other files will stil be available over HTTP, Qlik Sense will serve them.

A scenario I have seen used, is that a web developer buiilds a mashup, but then users (super-users ??) can modify files in it, like HTML, CSS or configuration files. In that case you can add a wbfolder.wbl to the extension containing just the files users should be able modify, but not other stuff, like libraries, possibly some JavaScript etc.

Qlik Sense Business Analyst certification

Yesterday I did the Qlik Sense Business analyst certification exam. Even though I’ve worked with the product since the development started and as a contractor since 2016 I have never done it before.

Certifications are good: they force you to make sure you cover more of the product. If you’ve worked with a product for a while, you tend to know parts of the product very well but might have missed some stuff completely. The certification forces you to fill in the gaps (or at least some of them..) and learn those features you never used in your projects. Perhaps you pick up something that comes handy in your next project. Personally I learned a lot on the Data Manager and quite a few visualizations I have never used.

The exam situation is pretty special. When I have taken certification exams previously (some time ago..) I have done it at testing centers, but this time I did it from home, probably the only possibility in the current Coron situation, and certainly the only one I would do. But then I had to clear my desk completely (never happened before), stay at my desk for the full exam (two hours) and keep my eyes on the screen, all very strange for me, who normally gets up and walk around a lot.

The exam is good: pretty difficult questions that really tests your knowledge. Taking it without having worked with the product, just from reading books and other material would be very hard, almost impossible.

It is available only in English. A tips when preparing, if you are not native english speaking, is to make sure you run Qlik Sense in english, to get the terminology right. You can do that by running Qlik Sense in the browser and adding ‘/language/en’ to the URL.

And I did make it, with a score of 76% (you need 58%). Feels good!

Qlik Sense Global object

Sometimes when you are working with the Qlik Sense APIs you need access to information that is not connected to the Qlik Sense document you are working with, but rather to the QIX Engine running.

You find the apis for this in the global object in the Capabilities APIs, or in the Global class in the Engine API. While the Capabilities APIs only include very few methods, the full list of calls in the Engine API is much longer.

Finding the Global object

While there is a method in the Capabilities API to get the global object, in most cases you should not use it. In fact if you are building an extension you never should. Instead you should use the global property on the app.

That is because the getGlobal() method will open a new web socket to the engine, while app.global property will use the web socket used for the app. There is not reason to open another web socket to the server when you already have one, and in an extension there always is one.

The only situation getGlobal() should be used is when you do not have an open web socket, like in a mashup before the app is opened. And in that case you might be better off with a REST call.

Chrome developer tools in Qlik Sense extension development (part 2)

As I wrote in Part 1 Chrome Developer tools is perhaps the most important tool for a Qlik Sense extension developer. Some important points in part 1:

  • you should turn caching off in the network tab, and keep developer tools open to make sure files are not cached
  • Qlik Sense client catches program errors in your extension, so if you need to see them temporarily check the box ‘Pause on caught exceptions’. You can’t keep this box checked all the time though.

In this post we will continue with some more useful stuff.

Checking web socket traffic

As you probably know communication with the Qlik Engine is over web sockets. You find the web socket in the Network tab, just click on WS and you will find it. Click on the websocket and then on messages, and you will see the communcation, something like this:

Lots of information, lots of transactions. Some things you should know:

  • messages from client to server has a dark background, messages from server to client a white background
  • all messages are in JSON format
  • if you click on a message it will be displayed in a format that’s easier to read in the area below
  • you can filter messages by typing in text in the field where it says ‘Enter regex..’

Connect request and reply

What you see in the web socket log is simply a list of all messages sent and received, in chronological order. Since commands are processed in paralell, the reply for a specific request might not at all be anywhere near the request. Request and reply are connected with the ‘id’ attribute (that’s a feature in the json rpc protocol, which Qlik Sense is built on). This means we can filter on the id attribute to find the reply for a specific request. If you try typing in “id”:1, in the filter box you will get request #1 and it’s reply:

Handling handles

So now we know how to find a request and it’s reply, but to really understand what’s happening we also need to understand handles.

Handles is a Qlik Sense specific concept, not part of Json RPC but an addition to it, made by Qlik. The QIX protocol is object-oriented, meaning that all commands are sent to an object. When you connect to Qlik Sense engine, the only handle available is the global handle, -1.

The first thing you do in most cases is call OpenDoc to open your Qlik Sense app (or Doc, as its called in the QIX documentation). If that succeeds you will get a handle (called qHandle) back in the reply. You can them send Doc commands to this handle (almost always handle 1). If you type in the filter “handle”:1, you will see all app commands used:

As you can see there is a lot of them. Most of them fetches or creates objects in the app, with GetObject or CreateSessionObject. You do not see the replies, only the requests, since the replies do not contain the handle. But if we instead filter by the id of one of the GetObject calls, (in my case id 13) we can see what happens:

As you can see in the reply there is a new qHandle in the reply, #5. So now we can filter on “handle”:5, and see what commands were sent to this object. If you want to know what handle is used for your extension, there is as far as I know no way to get that information from standard Qlik, but you can use my Chrome Extension Add Sense.

Find commands that invalidates object

Handles are used not only to refer to objects when you send commands, but also for invalidation. When you change selections (make a new selection, clear selection, apply bookmark etc) QIX engine tells you what objects are no longer invalid. It does this with the ‘change’ array (another addition to JSON rpc) which contains handles for the invalidated objects. So if I for example want to find commands that invalidated opbject 13, I can filter on “change”:\[.*13.*\] (note: a regex, since the handle can be at any position in the array):

We only get the replies for commands that invalidated the object, to get the actual command you need to use the id field to find the actual command and handle.

Chrome developer tools in Qlik Sense extension development (part 1)

Perhaps the most important tool for a Qlik Sense extension developer is Chrome developer tools. You can use any text editor to write your code, or even Qlik Sense dev-hub for simple extension, but you really need to use Developer Tools. Debug your code, check loading of javascript, HTML and CSS, inspect HTML structure and CSS rules – there is a lot you need to use them for. So good knowledge of them is essential.

Always run Qlik Sense in a browser

Qlik Sense Desktop is a good tool for your extension development in most scenarios. But you should never use the browser bundled with it for development. Instead use Chrome, and use http://localhost:4848/hub to access the hub. This means you will have the full developer tools available, and also that you will have an up-to-date Chrome installation.

While the browser bundled with Qlik Sense Desktop is built on the same codebase, it is not updated, but will contain an old version. It looks like the Chrome version bundled with Qlik Sense Desktop September 2019 is Chrome/47.0.2526.16 while my browser Chrome version is Chrome/78.0.3904.70, so Qlik Sense Desktop is way behind. BTW Qlik Sense Desktop also reports my Windows version as ‘Windows NT 6.1’ while Chrome proper reports the real value ‘Windows NT 10.0’.

So now that you are happily in Chrome you need to open the Developer Tools. There are different ways to do that:

  • press Shift+Ctrl i (for inspect, this is the method I use)
  • from Chrome menu in the top right corner, under ‘Other tools’

When doing extension development, always keep the console open. We’ll come to why later.

Disable caching

When doing extension development, you want Qlik Sense to allways load the latest version of your code. The browser normally tries to cache loaded files, and not request the file from the server again, it it already has it in memory. The first thing you should do is to disable this caching.It’s easy to do:

  • open developer console
  • switch to network tab
  • click the ‘Disable cache’ checkbox

The disable cache checkbox only affects the browser when the developer console is open. When the developer console is not open, the browser will still cache javascript files. This is why you should always open the browser console when your are working with extension development.

Check installed extensions (and mashups)

There is a lot of information in the network tab. You can se all network requests made by the Qlik Sense client, and since the client is load in chunks there will be a lot of them. There are also requests for images, CSS stylesheets and other requests. To help you filter the requests there is a bar:

Click on XHR and you will get a list of other calls. One of them is of special interest when you are doing extension development, the one called schema (url is really /qrs/extension/schema). Click on that one, and then on ‘Preview’ and you will get a list of all extensions and mashups installed on the system, including all data in the qext file:

This is a good place to check that your extension is installed, and what version (just remember to keep the version number in the qext file up to date). It’s also a good place to start when you are troubleshooting an extension you did not write.

Debugging the extension code

The ‘Sources’ tab is what you use for debugging. If the code is minified (many extensions are) you can click the curly braches in the bottom left corner to get a somewhat more readable format.

The toolbar in the upper right corner can be used to step through your code. The stop button will make the browser pause when an error occurrs, which is very useful to find bugs. Unfortunately it does not really work in extensions. That’s because Qlik Sense catches all extension errors (and silently ignores them).

So to have the browser pause on your errors, you need to check the box ‘Pause on caught exceptions’. If you do that, the browser will pause when an exception is thrown from your code. Unfortunately it will also break on some exceptions (in libraries) that are perfectly OK, so you can’t really turn it on always. Just turn it on when you think there might be an exception thrown from your code and then trigger a repaint, for example by resizin the window.

Check files in extension

The navigator on the left side in the sources tab shows you extension files in a tree structure. But it will only show you javascript files, so CSS and HTML files are not included. If you want to see those, switch to the network tab, and filter on the extension name.

Note that the qext file is not included. That’s because the qext file is not loaded by the client, the information is instead included in the schema.

and there’s more

Of course there is more, but I couldn’t cover everything in one post. Look out for part 2, available when I find the time to write it..

Qlik nebula.js – use in Qlik Sense

Previously I have written about Qliks new nebula.js framework: how to build a visualization and how you can use it in a web page. But what if you want to use it in the Qlik Sense built-in client?

Nebula.js is a new framework, built on an architecture that differs a lot from the old Qlik Sense client and has a new API. This means nebula.js-based visualization won’t work in Qlik Sense.

Add a Qlik Sense wrapper

So to run your visualization in Qlik Sense you need to wrap it in something that supports the Qlik Sense extension API. You also need to add the qext file. Luckily nebula.js can do that for you. Just type:

nebula sense

and nebulajs will wrap your visualization so it can be used as a Qlik Sense extension. You find the result in the [visualization]-ext directory, so for nova -table it will be nova-table-ext. Just copy the entire directory to your Extension directory (assuming you’re using Qlik Sense Desktop) and you will find your extension in the assets panel in sense:

and you can add your nebulaj.s visualization to your Qlik Sense app. You will also get a basic property panel, which works pretty well for the very basic nova-table.

Improving meta data

But the extension name is ‘nova-table’ and the icon used is the default extension icon and there is no description. You might be tempted to fix this in the generated qext file, but don’t do that! There is a better way. Start by getting some help by typing:

nebula sense -help

And you will get the following info:

This means we can add your meta info to a separate file and supply the filename as an argument to the ‘nebula sense’ command. So let’s create a file called meta.json and enter the following:

{
	"name": "Nebula table",
	"icon": "table",
	"description": "Nebula test table wrapped in a Qlik Sense extension"
}

Run the command:

nebula sense --meta meta.json

And copy the update -ext directory to youe Extension directory, overwriting the old version. You will see that the meta data has changed:

Working on the property panel

The wrapper also contains a default property panel, something you might want toimprove, for some visualizations you actually have to improve it to make the extension work. To do that we start with the default propertypanel. It’s in nova-table-ext/extDefinition.js, but it’s minified. To get it in a format that’s easier to work with we use the minify option to turn minification off (default is on):

nebula sense --minify false

And then copy the extDefinition.js file to the src directory. If you look at the file you can see a few things we could improve:

  • the settings section is commented out
  • features as export, exportData and snapshot are not supported

To make this work you also need to remove the define call the build script has wrapped around the definition (it will be added again when you build). So modify the file to:

const extDefinition = {
  definition: {
    type: 'items',
    component: 'accordion',
    items: {
      data: {
        uses: 'data',
      },
      settings: {
        uses: 'settings',
      },
    },
  },
  support: {
    export: true,
    exportData: true,
    snapshot: true,
    viewData: false,
  },
};

export default extDefinition;

And then run the build:

nebula sense --meta meta.json --ext src/extDefinition

Copy over the extension files again and you’ll se that you now have the ‘Appearance’ section (called ‘settings’ in the code) and export etc now works.