Kobo Toolbox related tutorials and tips

This page contains information on how to visualize data in Power BI, enable email notifications etc.

Repeating Groups

Think of repeating groups as something similar to a relational database, where multiple records in a repeating group correspond to a single record in the main query (one to many relationship)

1. Similar to our example with multiple choice questions, we will start with duplicating the original query and renaming it into something more descriptive (RepeatingGroup for example)

2. After this is done, select and remove all columns, except the repeating group and _id (at this point the repeating group still needs to be expanded, and is shown as a "list").

3. Click to expand the list to new rows. After this, you should get a different query (probably with a larger row count), where all records of that repeating group are shown

Make sure to properly format (right click -> change type) all relevent fields (whole numbers, decimals, dates etc.).

4. Next, close & apply the query editor, and head for the Relationships section of Power BI. You should be able to see 3 queries here. Double click the relationship between the main query (KoboData) and the repeating group (RepeatingGroup), and make sure that you select both from the cross direction filter drop down list. This will enable you to filter by values from the newly created query.

You can now use the values from the repeating group query for data analysis and visuals.

You could have multiple repeating groups in a single form. Just repeat these steps for all groups and make sure that proper relationships are set.

E-mail Notifications using Microsoft Power Automate (formerly Microsoft Flow)

It is possible to set up email notifications whenever there is a new form submission. To do this you need to log into Microsoft Power Automate. You can do this by signing in using your UNHCR domain username and password.

1. Once logged in, go to the My flows section. From the New flow dropdown menu, select Instant Cloud flow.

2. Next you will see a pop-up window with a selection of triggers. Select When a HTTP request is recieved and click Create

3. Click the green When a HTTP request is received box to unfold further settings. Enter the Request Body JSON Schema as displayed below. Add all fields needed for your flow, but please avoid sharing any private or sensitive data in this manner. Here, we are only using _id and form_title for demonstartion purposes. Field names should be indicated as in the Kobo form (note that this differs from question labels).

4. Now click on Next step. In the search box, select mail and then Send an email notification.

5. Now you will need to compose the email template. Under To: enter the recipient(s). This could be you/a colleague or the email of the respondee themselves. To add the respondee email, select the email field from the dynamic content pop-up. This will only be available if 'email' is included in the JSON schema from step 3. Also, enter the subject text you want displayed. Now you can "build" the email body by combining text and items from the schema (see example image). This is how the content of the email will appear.

Note that you can set up a similar flow for text messages using the service Twilio . To do so, you will need to set up a Twilio account and select Twilio as the operation in step 4.

6. After you click on Save go back to the When a HTTP request is recieved section. There should be a link generated there. Copy this link using the button next to it.

7. Now you need to go back to KoBo Toolbox, log in, and open the project that you want to enable e-mails for. Under the Settings menu of your project, go to REST Services and Register a new service.

8. Enter the Name of the new service ("Microsoft Flow" or "Email" for example), and in the Endpoint URL section, paste the URL you copied from Microsoft Power Automate. Under Select fields subset, enter the fields that you wish to post to Microsoft Power Automate. You will need to click "enter" or "tab" on your keyboard after entering each field name. In our example we used _id, form_title, email and _5_Phone_number but you can add additional fields/questions from the form. Please note that you are posting these fields outside of Kobo Toolbox. Be careful when working with sensitive data.

KoBo Image Downloader

The KoBo image downloader is a custom-developed applicaiton that can help you easily download all images (and other files) associated with a certein project.

1. You can download the application here.

2. Export the contents of the .zip file.

3. Run Kobo image downloader.exe.

4. Click on Select folder to select where the KoBo images/files will be downloaded.

5. Enter your KoBo Toolbox Username and Password.

6. Click on 2. Load Projects (might take some time, depending on the number of projects).

7. Select the project you need from the drop-down list to the right of the button.

8. You can querry the records as shown in the example. This is usefull for large data-sets.

9. Click on Load JSON to retreive the data (might take some time).

10. Click on Load Images to download the images/files to the folder you previously selected.

API Querying

The Kobo Toolbox API v2 allows data querying using a URL. This is useful when working with large datasets in order to reduce loading time, and for getting around JSON size restrictions.

1. Start by going to your Kobo account to locate the form ID (uuid). Select the project you wish to query from your list of deployed forms. From here you can get the form ID (uuid) from the URL. The URL of the page you are currently on will follow this format:

https://www.kobo.unhcr.org/#/forms/{uuid}/landing

Copy the string of characters between the two slashes. See below for an example.

2. Paste a URL with the following format into your browser to go to the API endpoint for this form. Insert the form ID from step 1. Please note that the curly brackets should not be included.

https://kobo.unhcr.org/api/v2/assets/{uuid}/data/

From the GET dropdown menu, select the data type you need; most commonly json.

Your browser will return the submitted data in a json format as such:

3. To filter the JSON data, you can add the query strings directly to the URL. You can query by various items (e.g. ID, date...)

4. You can filter by the submission ID using the example in the image below (in the example, we added &query={"_id":"13010440"} to the end of the URL). This is possible to do without using querying, but can also filter all submissions, for example, "greater than" the selected ID (check the "submission date" example below).

5. The next example shows how to query all submitions after (greater then) a certain date (in the example, we added &query={"_submission_time":{"$gt":"2021-03-17T09:39:19"}} to the end of the URL).

6. You can also limit the number of records displayed in the results (in the example, we added &limit=1 to the end of the URL).

These are just some examples of fields that you can query by, and of ways to do it. Visit the Mongo DB manual for additional examples on how to use querying. Please be careful using data querying through the API, when doing data analysis and visualization.

For other use cases for the API (e.g. querying form metadata or user assets) please see the following resources:

Kobo Toolbox API documentation

Kobo Community API v2 Examples

Select Item from CSV File - Enketo

Kobo toolbox allows you to upload a CSV file (or multiple files) containing a list of items. Instead of relying on the choices tab in the XLS form, you can upload a sepparate CSV file, and make a referrence to that file in the form. This is especially useful when working with long lists with more than 500 items (like locations or proGres ID numbers). Cascading select still works when using external CSV files.

1. When creating the XLS form, under type, instead of using the normal select_one with the list name, you would use select_one_from_file filename.csv. The filename.csv is a sepparate file that you will be uploading to KoBo.

2. The CSV file should contain the name and label columns, similar to a normal list.

3. After you have both the XLS form and CSV file, you should upload the XLS form to KoBo toolbox as normal, and deploy it. You can find examples of both the XLS form and CSV file here. Please note that in our CSV file example, we also have a third column registration_id because we will be using cascading select. The third column is not necessary for select_from_file to work.

4. Once the form is deployed, you need to upload the CSV file. Go to the settings section of the project, and click on media in the side-menu. After that, simply click on Add Documents, select the file(s) you wish to upload, and click on the Upload button.

5. After the file has been uploaded, you should see the lists in your deployed form similar to the example here

Important notes: This functionality is currently only available in deployed forms. This is not available when previewing forms, or in the form builder. This functionality is only available on Enketo (ODK/KoboCollect is currently not supported). Also, the CSV file that you are uploading is not encrypted, so please do not use it for storing sensitive data.

Search functionality - ODK/KoboCollect

For referencing lists in CSV files when using ODK/Kobocollect, you can use the search function. The process is quite similar to using select_one_from_file (see here), but this method only works in ODK/Kobocollect, and not in Enketo.

You can download the XLS form and CSV file examples here.

Once you deploy the project and upload the CSV file, you need to download the form in ODK/KoboCollect in order to test it.

Retrieving KoBo records for editing using Power BI

Using Power BI, it is possible to access KoBo records for editing, without needing to go through the KoBo back-end. To do this, you need a table visual to display the records, and a custom column to generate the link to the record.

1. To create the custom column, you need to open your report (that is using KoBo Toolbox as a data source) in Power BI Desktop, go to query editor and click on Add Columns -> Custom Column. You will be using 2 columns generated by KoBo [_xform_id_string] and [_id]. Make sure that these columns have not been removed in Power BI, or that the names have not been changed (a prefix added for example). If the name has been changed, you will have to reference them using the new name.

2. This is the formula that you need to paste into the Custom column formula area (copy the entire formula):
"https://im.unhcr.org/kobosupport/link/index.php?xformid=" & [_xform_id_string] & "&recordid=" & Text.From([_id])

[_xform_id_string] refers to the form id. The field-name is the same for every form. (make sure that the column has not been removed from the Power BI report, and that the name has not been changed)

[_id] refers to the record id. The field-name is the same for every form. (make sure that the column has not been removed from the Power BI report, and that the name has not been changed)

The rest of the link refers to a PHP script that will prompt you to log in, pass the IDs to KoBo, and redirect you to the record.

3. Next, you need to return to the dashboard area, and create a table visual with the columns you wish to display, and the custom column that you just created.

4. By clicking on the custom column in the fields area, and going to the modeling tab in the top menu, you can change the formatting of the column by selecting Data Category -> Web URL.

5. In the formatting area of the visual, under values, you can set URL icon to on, which will change the link into an icon.

6. That's it. Now, you just need to click on the link, which will take you to a log-in screen where you need to enter your kobo credentials. After that, you will be able to edit the record.

Security Tips and Best Practices

Here you can find some of the best security practices related to account creation, project sharing, project deployment, connecting to the API and others.

1. Do not use your personal KoBo account as a main account for an operation/project. Kobo does not allow changing the ownership of projects, so you should always try to have an operation level or project level account. You can create multiple accounts using a single e-mail address which you can later change in the account settings. Once the email address is changed, the new email will be used to retrieve the password using the reset password link.

2. Never share the credentials of the project owner account. This account has full access to the to submit, view, edit and validate the data, and make changes to the form. In case you need to share the project (data or form), be sure to provide the minimum set of permissions required (for example, if the user needs to view the data or use it for data visualization in Power BI, only select "View submissions". If the user is an enumerator, only select "Add submissions").

Be mindful that if a user has the "view data" permission, that user can see the entire data of that project.

3. By default KoBo allows anonymous submissions, and anyone with the enketo link can submit data to the project. To make sure that only users with "Add submissions" permission can submit, you need to check the require authentication to see forms and submit data option in the account settings. This is an account-level option.

4. If you need to record the accounts that are submitting data (only when "require authentication" is enabled), check username in the "metadata" section of the "layout & settings" menu when editing the form. Based on your requirements you can have individual enumerator accounts, or you can have multiple persons using the same account (like partner accounts for example). You can create multiple accounts using a single email address, and please don't share the project owner account credentials.

5. Do not check the "share data publicly" option in the project sharing settings as this is not a requirement for connecting Power BI to KoBo. Also, when using Power BI, be sure to remove all sensitive fields (like names, addresses etc.) in the query editor. Just because the sensitive information is not displayed in the charts, users might still be able to access it using right click -> view records unless the columns are completely removed.

Setting up ODK Collect

When doing data collection in areas with limited internet connectivity, it is reccommended that you use ODK Collect, which can be downloaded on Android devices from the Google Play Store. You can find instrucitons on how to configure ODK Collect here.

Useful Links

XLS Forms

Kobo Toolbox API documentation

KoBo Toolbox Community Forum

koboloadeR - R package for conducting data discovery and analysis