Metadata syncing service
The “metadata syncing service” is a mechanism to synchronize metadata between reporting instances and the Apliqo UX Content Store. The scope of the synchronization is quite limited and includes only:
- Principal name of all elements in the }Clients dimension and the }TM1_DefaultDisplay value alias
- Principal name of all elements in the }Groups dimension and the }TM1_DefaultDisplay value alias
- Members belonging to each group
- Principal name of all elements in the }Cubes dimension
- Principal name of all elements in the }Dimensions dimension
- The Default_Member of each hierarchy as defined in the }HierarchyProperties cube
By default all users and groups are synced. For customers with very large number of groups and/or users, as a performance optimization after initial synchronization the sync can run in “content store lead” mode where only groups which are flagged in the content store to by synced have their members read and updated in the Content Store.
It is possible to have a fully functioning Apliqo UX application without ever using the metadata syncing service. However, if you do use the service you will get up and running with your application faster, avoid manual maintenance tasks, and be able to take advantage of more advanced features.
The primary reason for being of the metadata syncing service is to reduce manual maintenance! Specifically to eliminate the need to set up users and groups in the Content Store when all users, and presumably all needed role definitions and group memberships are already maintained elsewhere in your primary TM1 server instances (“reporting instances” from the perspective of Apliqo UX).
The metadata syncing service can completely automate user and group creation for you, including group memberships, providing that the definition of the group memberships exist on another server which is accessible to Apliqo UX. In addition, by syncing dimension and cube metadata you can define default context elements for key “point of view” dimensions which will be applied to the setting service when users log in, and you can apply default row and column layouts to cubes which facilitates faster report building by your power users.
The syncing service is provided by the UXToolkit utility (uxtoolkit.exe) which is included with the installation of the Apliqo Application Server. The utility is a windows command line executable based on tm1Py and is part of the UX installation since version 2.5.2. The utility is also available in earlier versions of Apliqo UX as a python script. This requires significantly more setup as the correct version of python, tm1Py and other module dependencies must be installed manually.
Inside the root directory of the apliqo server you should see a “py” folder which contains
The ReadMe document contains a change log for the utility and detailed technical documentation of the parameters which can be passed via command line. This document should contain sufficient detail for regular usage and standard use cases. However, please refer to the ReadMe document in the installation directory for the definitive list of available features and available parameters with your version.
The settings.json file is very important. This file contains the instructions for the utility to log into the ContentStore and the location of the instances.json file associated with your Apliqo UX application.
The password field can contain plain or encrypted text. We recommend encrypting the stored password using the utility’s “store” command (described below).
pathToInstances can be either a full path or a relative path based on the location of the settings.json file.
Note: all parameters contained in the settings.json file can be overridden by passing runtime parameters in the command line. For any omitted parameter the value from the settings file will be used. If no path to the settings file is passed, then the UX Toolkit will search the directory where it is located.
There are a number of command line arguments which can be passed to the UX Toolkit regardless of what kind of operation it is performing. Arguments which are specific to a defined "task mode" are explained in the following section on each mode.
path : the path to the settings.json file. If omitted then the toolkit assumes the settings file is located in the same directory as the uxtoolkit.exe itself e.g.
uxtoolkit.exe --path "C:/MyFiles/settings.json"
instances_path : the path to the instances.json file. If omitted then the toolkit will use the pathToInstances value from the settings.json file
uxtoolkit.exe --instances_path "C:/ApliqoServer/webapps/UX_Demo/WEB-INF/instances.json"
cs_cam : the CAM Namespace for authentication to the ContentStore. If omitted then the toolkit will use the value from the settings.json file
uxtoolkit.exe --cs_cam "AzureAD"
cs_user : the user ID for authentication to the ContentStore. If omitted then the toolkit will use the value from the settings.json file. If using CAM then the user ID must be the user ID used when authenticating to CAM. For all other authentication modes it should be the element name from the }Clients dimension
uxtoolkit.exe --cs_user "Admin"
cs_password : the password for the specified user for authentication to the ContentStore. If omitted then the toolkit will use the value from the settings.json file. Can be passed either as plain text or encrypted string
uxtoolkit.exe --cs_password "apple"
key : the encryption key for password encryption/decryption. If omitted then the toolkit will use a built-in default key
uxtoolkit.exe --key "DoGoodTM1"
debug : instructs the utility to run in debug mode and write debug log to the console
uxtoolkit.exe --debug true
The above parameters apply to all operating modes (store, transfer, createattrs, clean) and must be placed BEFORE the positional arguments in the command line string.
Warning! If you choose to supply a custom key when storing the password then you must also supply the same key value on all subsequent runs of the utility.
In addition to the basic parameters, the utility accepts one of 4 alternate positional arguments which control the mode of operation. These are:
- store : stores username, password, CAM Namespace, pathToInstances into the specified settings.json file
- clean : performs a "clean up" of a specified instance or all instances available in the instances.json file. Objects that are deleted from the system are all redundant objects from Apliqo UX 1.x versions (e.g. objects with }APQ C3 Canvas* naming)
- transfer : transfer metadata to the Content Store. Which data is transferred depends on supplied modifying parameters (-initialize, -security, -data, -all)
- createattrs : create attributes in a selected instance or in all instances based on Create flag in }APQ UX Instance Dimension Attribute cube
The modes of operation and additional modifying parameters for each more are described in the next section.
The utility has 4 different modes of operation or tasks which it can perform.
Store mode is designed as an assistance to initial configuration and writes (stores) values to the settings.json file. In all other modes the settings file is read by the utility and provides default parameters for connecting to the Content Store and the reporting instances. The Toolkit will also attempt to authenticate to the Content Store with the credentials in (or being written to) the settings.json file. If connection is unsuccessful the routine will abort with error and not update the file. If successful then in addition to the settings file being updated the authentication credentials will also be stored in the }APQ UX Instance Attributes cube in the Content Store.
This is most likely a once off configuration task which will be run once manually. There would only be a need to run again if the password changes for the background admin account or the location of the webapp is changed.
Of course you can edit the settings.json file directly in Notepad++ or Visual Studio Code but to store the connection password as an encrypted string then you must use the store function. Store mode accepts the following modifying parameters:
cam : the CAM Namespace for authentication to the ContentStore. If omitted then the toolkit will not update the CAMNamespace value in the settings.json file
uxtoolkit.exe store --cam "AzureAD"
user : the user ID for authentication to the ContentStore. If omitted then the toolkit will not update the userName value in the settings.json file. If using CAM then the user ID must be the user ID used when authenticating to CAM. For all other authentication modes it should be the element name from the }Clients dimension
uxtoolkit.exe store --user "Admin"
password : the password for the specified user for authentication to the ContentStore. If omitted then the toolkit will not update the password value in the settings.json file. Can be passed either as plain text or encrypted string
uxtoolkit.exe store --password "apple"
instances_path : the path to the instances.json file for the webapp. If omitted then the toolkit will not update the pathToInstances value in the settings.json file
uxtoolkit.exe store --password "apple"
instance : a specific instance from instances.json. If an instance is specified then the connection credentials for the instance will be updated in the }APQ UX Instance Attributes cube only and the settings.json file will not be edited. If omitted then the toolkit will update ContentStore credentials in the settings.json file
uxtoolkit.exe store --password "apple"
Note: that the parameters for store; cam, user, password are different and not to be confused with the similar standard parameters in that they specify values to be written to the settings file whereas the standard parameters; cs_cam, cs_user, cs_password specify values to use in preference to the default values in the settings file (i.e. the settings file is ignored if these parameters are supplied).
E.g. to encrypt the password and store directly into settings.json file:
uxtoolkit.exe --path "C:/MyFiles/settings.json" store --user "Admin" --password "apple"
From UX version 2.5.2 there is now an administration app provided to make this process easier (you don’t have to use command line unless you really want to!)
In the background the built-in content Store process }APQ.UX.Toolkit.Run.Store provides a wrapper to call the uxtoolkit.exe program via ExecuteCommand.
Transfer mode is the main work of the UX Toolkit utility. Transfer mode connects to each specified reporting instances and reads the }Clients, }Groups, }Dimensions and }Cubes metadata and transfers this information to staging cubes in the Content Store.
Transfer mode accepts the following modifying parameters:
initialize : reads the instances.json file and checks all instances are set up in the }APQ UX Instance dimension. Any instance in the logonInstances array will have the credentials for ContentStore connection duplicated to the instance entry in the }APQ UX Instance Attributes cube
uxtoolkit.exe transfer --initialize
security : reads the instances.json file and authenticates to each instance and writes the contents of the }Clients and }Groups dimensions as well as the }ClientGroups cube back to the ContentStore
uxtoolkit.exe transfer --security
data : reads the instances.json file and authenticates to each instance and transfers security and then writes the contents of the }Dimensions and }Cubes dimensions back to the ContentStore. (E.g. performs security step, then also copies cube & dimension names)
uxtoolkit.exe transfer --data
all : performs the initialize and data steps
uxtoolkit.exe transfer --all
cs_lead : Boolean default=false. If true then only groups previously marked with the ContentStoreImport flag will be queried for their members to update to the Content Store
uxtoolkit.exe transfer --data --cs_lead true
Positional parameters dictating the mode of operation can of course be combined with global parameters which need to be specified before the mode n the command line. For example:
instance : a specific instance from instances.json. If an instance is specified then the Toolkit will connect to and copy metadata only for the specified instance. If omitted then the toolkit will cycle through all instances in the instances.json file (including the Content Store itself)
uxtoolkit.exe instance "TM1PROD" transfer --security --cs_lead true
Typically "initialize" is a once off operation. Likewise, once the list of dimensions and cubes has been synced with the "data" step this would not usually need to be run again. However the "security" sync would typically be done on a regularly scheduled basis (e.g. nightly) in order to keep users and (synced) group memberships in sync between the reporting instance and Content Store and moreover to keep the Content Store maintenance free.
Scheduling such a task is a simple process with the built-in content Store process }APQ.UX.Toolkit.Run.Transfer which provides a wrapper to call the uxtoolkit.exe program via ExecuteCommand.
From UX version 2.5.2 there is now an administration app provided to make administering the metadata syncing easy.
In the application below the "Fetch Users & Groups" button (marked step #1) runs the process }APQ.UX.Toolkit.Run.Transfer as described above, which calls uxtoolkit.exe in transfer -security mode
This syncs the reporting instance security model to the }APQ UX Instance ClientGroups cube. The UX administrator then has the option to flag whether to automatically import users and groups into the content store. Typically you would import all users (step #2 left). However, you would probably opt to only import groups to the content store which are relevant to Content Store security (step #2 right). (You may decide not to import any groups.)
The button "Sync to Content Store" (step #3) will run the process }APQ.UX.Security.ClientGroups.Import.FromInstances which will create all users marked to be imported and likewise for all groups marked to be imported will match the group memberships. Note that this process is already include in the daily maintenance chore. However you could also run this directly after any scheduled UX Toolkit sync in order to fully sync the Content Store security to the reporting instance in the lead.
Many advanced features in Apliqo UX can be easily controlled via attributes. Some attributes which control features in Apliqo UX are standard attributes which have universal properties in other Planning Analytics UIs like Caption or Format. However, there are other attributes which have a special purpose in Apliqo UX. For example:
- Color : if an attribute is set with a standard color name or HexCode this will be recognized by Apliqo UX and all data series will be automatically plotted in the selected color consistently in all reports
- ibcs-class : this attribute can be used in version or scenario dimensions but also in continuous time dimensions for consistent formatting in both table column headers and chart series. The use case is similar to the Color attribute, for example to ensure that Budget, Actual and Last Year are always formatted consistently.
- CellType : This attribute is most commonly applied to measure dimensions used on column or rows in Apliqo UX applications. Common cell types are "Checkbox" which applies a table cell class of checkbox (as used on this screen) and "Date" which signals the cell value should be entered via data picker modal.
- Group : If a group attribute is set this is used on column dimensions of views to automatically "nest" elements and allow whole groups to be displayed or hidden with a single click.
- Renderer : The renderer attribute is to support "Scorecard Metrics" KPI reports with the same formatting as supported by standard IBM Cognos interfaces.
Using the createattrs mode switch in the UX Toolkit we can easily create such attributes on the target reporting instance.
In the background the }APQ.UX.Toolkit.Run.CreateAttrs wrapper process calls UX Toolkit. See the section on the built-in administration app for more information.
The UX Toolkit has one final function which is to assist in removing redundant objects on reporting instances from Apliqo UX 1.x applications. (That is pre-Content Store versions of UX).
By running the Toolkit in clean mode all "C3 Canvas" objects will be automatically removed from the system.