Domino Upgrade

VersionSupport end
Upgrade to 9.x now!
(see the full Lotus lifcyle) To make your upgrade a success use the Upgrade Cheat Sheet.
Contemplating to replace Notes? You have to read this! (also available on Slideshare)


Other languages on request.


Useful Tools

Get Firefox
Use OpenDNS
The support for Windows XP has come to an end . Time to consider an alternative to move on.

About Me

I am the "IBM Collaboration & Productivity Advisor" for IBM Asia Pacific. I'm based in Singapore.
Reach out to me via:
Follow notessensei on Twitter
Amazon Store
Amazon Kindle
NotesSensei's Spreadshirt shop
profile for stwissel on Stack Exchange, a network of free, community-driven Q&A sites


Serving Single Page Applications with Domino

Single Page Applications (SPA) are all the rage. They get developed with AngularJS, ReactJS or {insert-your-framework-of-choice}. Those share a few communialities:
  • the application is served by a static web server
  • data is provided via an API, typically reading/writing JSON via REST or graph
  • authentication is often long lasting (remember me...) based on JWT
  • authentication is highly flexible: login with {facebook|google|linkedin|twitter} or a corporate account. Increasingly 2 factor authentication is used (especially in Europe)
How does Domino fit into the picture with its integrated http stack, authentication and database? The answer isn't very straight forward. Bundling components creates ease of administration, but carries the risk that new technologies are implemented late (or not at all). For anything internet facing that's quite some risk. So here is what I would do:
Red/Green Zone layout for Domino


Domino, Extlib, GRUNT, JSON and Yeoman



With a few tweaks and clever setup, you can have web developers deliver front-ends for Domino without ever touching it.
Contemporary web development workflows separate front-end and back-end through a JSON API and HTTP (that's 21st century 3270 for you). The approach in these workflows is to treat the webserver as source of static files (HTML, CSS, JS) and JSON payload data being shuffled back and forth. This article describes how my development setup makes all this work with Domino and Domino designer.

The full monty

The front-end tools I use are:
  • Node.js: a JavaScript runtime based on Google's V8 engine. It provides all runtime for all the other tools. If you don't have a node.js installation on your developer workstation, where were you hiding the last 2 years?
  • Bower: a dependency manager for browser files like CSS, JS with their various frameworks. You add a depencency (e.g. Bootstrap) to the bower.json file and Bower will find the right version for you
  • Grunt: a task runner application, used for assembly of web sites/applications (or any other stuff). Configured using a Gruntfile.js: it can do all sorts of steps like: copy files, combine and minify, check code quality, run tests etc.
  • Yeoman: A application scaffolding tool. It puts all the tools and needed configurations together. It uses generators to blueprint different applications from classic web, to reveal application to mobile hybrid apps. The site lists hundreds of generators and you are invited to modifiy them or roll your own
  • GIT: the version control system
The list doesn't include one or the other editor, one or the other IDE or version control client, since theses choices don't influence the workflow steps or setup. Pick what you like.
My general directory layout starts with a project folder that has entries for code, documentation, nsf and a misc. folder. I only put the code folder into the GIT version control.
Seperation of concerns for frontend and backend development
The front-end developer doesn't need any of the Domino components available to create user interface and interaction models. The back-end developer will use exclusively the REST controls from the Domino XPages Extension library (which is part of a default current Domino server install). The two directory structures are linked in a GIT project, but that isn't mandatory. One step that made my live much easier: take the dist directory and link it to WebContent. This saves me the step to copy things around. The fine-tuning of the setup is where the fun starts.


email Dashboard for the rest of us - Part 2

In Part 1 I introduced a potential set of Java interfaces for the dashboard. In this installment I'll have a look on how to extract this data from a mail database. There are several considerations to be taken into account:
  • The source needs to supply data only from a defined range of dates - I will use 14 as an example
  • The type of entries needed are:
    • eMails
    • replies
    • Calendar entries
    • Followups I'm waiting for
    • Followups I need to action
  • Data needs to be in details and in summary (counts)
  • People involved come in Notes addresses, groups and internet addresses, they need to be dealt with
Since I have more than a hammer, I can split the data retrieval into different tooling. Dealing with names vs. groups is something best done with LDAP code or lookups into an address book. So I leave that to Java later on. Also running a counter when reading individual entries works quite well in Java.
Everything else, short of the icons for the people, can be supplied by a classis Notes view (your knowledge of formula language finally pays off).


email Dashboard for the rest of us - Part 1

One of the cool new features of IBM Verse is the Collaboration Dashboard. Unfortunately not all of us can switch to Verse overnight, so I asked myself: can I have a dashboard in the trusted old Notes 9.0 client?
Building a dashboard requires 3 areas to be covered:
  1. What data to show
  2. Where does the data come from
  3. How should the data be visualised, including actionable options (that's the place we preferences between users will differ strongly)
For a collaboration dashboard I see 3 types of data: collaborators (who), summary data (e.g. number of unread eMails) and detail data (e.g. the next meeting). Eventually there could be a 4th type: collections of summary data (e.g. number of emails by category). In a first iteration I would like to see:
  • Number of unread eMails
  • Number of meetings left today
  • Number of waiting for actions
  • Number of action items
  • List of top collaborators
  • List of todays upcoming meetings
  • List of top waiting for actions
  • List of top action items
I'm sure there will be more numbers and list coming up when thinking about it, but that's a story for another time.


XPages XML Document DataSource - Take 2

For a recent project I revisited the idea of storing XML documents as MIME entries in Notes - while preserving some of the fields for use in views and the Notes client. Jesse suggested I should have a look at annotations. Turns out, it is easier that it sound. To create an annotation that works at runtime, I need a one liner only:
@Retention(RetentionPolicy.RUNTIME) public @interface ItemPathMappings { String[] value(); }
To further improve usefulness, I created a "BaseConfiguration" my classes will inherit from, that contains the common properties I want all my classes (and documents) to have. You might want to adjust it to your needs:
ackage com.notessensei.domino;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
 * Common methods implemented by all classes to be Dominoserialized
@XmlRootElement(name = "BaseConfiguration")
public abstract class BaseConfiguration implements Serializable, Comparable<BaseConfiguration> {
    private static final long serialVersionUID = 1L;
    @XmlAttribute(name = "name")
    protected String          name;

    public int compareTo(BaseConfiguration bc) {
        return this.toString().compareTo(bc.toString());
    public String getName() {
    public BaseConfiguration setName(String name) { = name;
        return this;
    public String toString() {
        return Serializer.toJSONString(this);
    public String toXml() {
        return Serializer.toXMLString(this);

The next building block is my Serializer support with a couple of static methods, that make dealing with XML and JSON easier.
package com.notessensei.domino;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;
import javax.xml.bind.Unmarshaller;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;

 * Helper class to serialize / deserialize from/to JSON and XML
public class Serializer {

    public static String toJSONString(Object o) {
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        try {
            Serializer.saveJSON(o, out);
        } catch (IOException e) {
            return e.getMessage();
        return out.toString();

    public static String toXMLString(Object o) {
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        try {
            Serializer.saveXML(o, out);
        } catch (Exception e) {
            return e.getMessage();
        return out.toString();

    public static void saveJSON(Object o, OutputStream out) throws IOException {
        GsonBuilder gb = new GsonBuilder();
        Gson gson = gb.create();
        PrintWriter writer = new PrintWriter(out);
        gson.toJson(o, writer);

    public static void saveXML(Object o, OutputStream out) throws Exception {
        JAXBContext context = JAXBContext.newInstance(o.getClass());
        Marshaller m = context.createMarshaller();
        m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
        m.marshal(o, out);

    public static org.w3c.dom.Document getDocument(Object source) throws ParserConfigurationException, JAXBException {
        DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
        DocumentBuilder db = dbf.newDocumentBuilder();
        org.w3c.dom.Document doc = db.newDocument();
        JAXBContext context = JAXBContext.newInstance(source.getClass());
        Marshaller m = context.createMarshaller();
        m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
        m.marshal(source, doc);
        return doc;

    public static Object fromByte(byte[] source, Class targetClass) throws JAXBException {
        ByteArrayInputStream in = new ByteArrayInputStream(source);
        JAXBContext context = JAXBContext.newInstance(targetClass);
        Unmarshaller um = context.createUnmarshaller();
        return targetClass.cast(um.unmarshal(in));

The key piece is for the XML serialization/deserialization to work is the abstract class AbstractXmlDocument. That class contains the load and save methods that interact with Domino's MIME capabilities as well as executing the XPath expressions to store the Notes fields. The implementations of this abstract class will have annotations that combine the Notes field name, the type and the XPath expression. An implementation would look like this:
package com.notessensei.domino.xmldocument;
import javax.xml.bind.JAXBException;
import lotus.domino.Database;
import lotus.domino.Document;
import lotus.domino.NotesException;
import lotus.domino.Session;
import com.notessensei.domino.ApplicationConfiguration;
import com.notessensei.domino.Serializer;
import com.notessensei.domino.xmldocument.AbstractXmlDocument.ItemPathMappings;

// The ItemPathMappings are application specific!
@ItemPathMappings({ "Subject|Text|/Application/@name",
					"NumberOfActions|Number|count(//action)" })
public class ApplicationXmlDocument extends AbstractXmlDocument {

    public ApplicationXmlDocument(String formName) {

    public ApplicationConfiguration load(Session session, Document d) {

        ApplicationConfiguration result = null;
        try {
            result = (ApplicationConfiguration) Serializer.fromByte(this.loadFromMime(session, d), ApplicationConfiguration.class);
        } catch (JAXBException e) {
        try {
        } catch (NotesException e) {
            // No Action Taken
        return result;

    public ApplicationConfiguration load(Session session, Database db, String unid) {
        Document doc;
        try {
            doc = db.getDocumentByUNID(unid);
            if (doc != null) {
                ApplicationConfiguration result = this.load(session, doc);
                return result;

        } catch (NotesException e) {

        return null;


View driven accordeon

The Extension Library features a Dojo Accordion control. In the sample application the panels and their content are created static using xe:basicContainerNode and xe:basicLeafNode. A customer asked: "Can I populate the accordion using view entries?"
Working my way backwards I first designed the format for the SSJS I need for the menu and then how to generate this format from an categorized view. The format that suited this need looks like this:
   { name : "Level 1"; items : ["red 1":"blue 1":"green 1"]},
   { name : "Level 2"; items : ["red 2":"blue 2"]},
   { name : "Level 3"; items : ["red 3":"green 3"]}

In a live application, the items rather would be objects with a label and an action (items : [{"label": "red 2", "action" : "paintred"}:{"label" : "blue 2", "action" : "playBlues" }]) or URL, but for the demo a string is sufficient. I created a accordeon with a xe:repeatTreeNode containing a xe:basicContainerNode as sole child element (which would repeat).
This child then contains one xe:repeatTreeNode that would again contain one xe:basicContainerNode as sole child (which again would repeat).
In theory that all looks brilliant, except the inner repeatTreeNode would not get access to the repeat variable from the parent basicContainerNode.
This has been accnowledged as a bug and is tracked as SPR # MKEE9SKENK
(I'm, inside IBM, outside the development team, the undisputed XPages SPR champion).
So a little hack was necessary to make that menu work.


Document dependent inline forms

In Notes client applications we are used to opening any document from a view and get it redered with the form specified in the Form (or the view's form formula). XPages behaves in a similar fashion when using the view control. However both approached open a new page (a tab in the Notes client, replaceing the current window in a browser).
Modern applications however expect a Single page application behavior, where only parts of the screen are replaced with new information. The XPages Extension library makes this easy, by providing the InPlace Form custom control.
Looking at the source code of the example page, it looks easy enough. However a real life application needs a few more considerations:
  • I want to open a (different) form depending on the current document
  • That form needs to be stored in its own control, so it can be reused
  • It is OK to enumerate the possible forms in the page (not fully flexible)
  • When saving an edit the form needs to disappear
With the help of the devine Maire Kehoe, this is the solution I came up with:
<xp:eventHandler event="onclick"
    submit="true" refreshMode="partial"
	<xp:this.action><![CDATA[#{javascript:var formId = getSubform(aViewEntry);

The code above is the event handler that goes into the link or button that triggers the edit action. In my case inside a repeat based on a view. The code to find the form is simple.
function getSubform(ve:NotesXspViewEntry) {
	// Put real logic here, like looking at the form
	// var form = ve.getDocument().getItemValueString("Form");
	// return "inPlace"+form;
	var fruit = ve.getColumnValue("Subject");
	var subform = ("Durian"===fruit) ? "inPlaceDurian" : "inPlaceFruit";
	return subform;


Mail Merge with XPages

Being able to have individualized letters based on a template was one of the drivers to make Word processors popular. In the age of mass-communication of one. This tasks falls no longer to the printer, but your eMail processor. For a complete solution, check out Chris Toohey's excellent Mailer application, that is yours for USD 5.00 only.
I was wondering what it would take to build something similar, minus the address management, in XPages. I defined a few constraints:
  • I don't want to store the address list or the raw message
  • I need to be able to send images
  • I need to be able to fine tune the HTML
  • No sending of attachments required
  • Message needs to be individual using parameters (in the Body only), list of parameters will be flexible supplied together with the eMails as CSV data
So I created the MergeManager Java bean. Along the way I made a few experiences:
  • When bound to a bean the XPages RichText control wants a data type of
  • The IBM Image upload doesn't work with a bean bound RichText control, but dragging and dropping works. I got rid of the button with a single line of code:
    config.removeButtons = 'IbmImage';
  • When adding the Dropdown for the variables, a simple change allowed me to show the View source button:
    {"name" : "mustache", "items" : ["mustache","Source"] });
  • I reused parts of Toni's eMail bean, which I stripped of the attachment function and added direct deposition into the Works like a charm
  • There are lots of loose ends to consider: using the OpenNTF Domino API to gain easy access to theads to send the messages in the background would be top on the list. Allow template storage, different messages for HTML and Text (and EE), eMail preview some of the others. But that's a story for another time
The result is a reusable bean for custom mass eMails.


Application Migration vs. XPage enablement

In a recent customer discussion a claim was made: "If Notes client application don't automagically can be converted into XPages applications, then we very well can migrate to another platform, it is just the same". An interesting claim, I'll subject it to a reality check. In any case it is a good idea to revisit your investment in your existing applications first.

XPages Enablement

Gradual availability of new functions
  • A reasonable estimate for the required effort is up to 30% of the initial effort. It very much depends on the quality of the existing applications and familiarity and skills of your team
  • There is no data migration, so existing users using a Notes client can continue to use the application until you retire the Notes clients
  • You can look at your application from a functionality and user perspective. A typical first XPages application is an approval module that spans across all your existing applications and provides a single view into anything that needs approval. A next level is forms and tools for mobile personell
  • Domino's security model works in XPages too, so no changes are required
  • If a function in an XPage shows a bug, it doesn't stop executing the function, since a user can fall back to a Notes client until the bug is fixed
  • Mail notifications just work, it's Domino. Also you can build Attention Central
So in a nutshell: low risk, incremental availability, no data migration, freedom to innovate


Providing user information in JSON

In the MUSE project we encountered the need to retrieve user information in JSON format. Easy done one would think. The trouble starts, when you have multiple directories and you need reasonable speed. Sometimes falling back to @Fomulas gives you what you need, fast and easy. @NameLookup knows where to look and you don't need any extra configuration. A simple call to an form will give you all you need: https://yourserver/somedb.nsf/namelookup?Readform for yourself or add &User=John Doe for any other user. This will return:
"QueryName": "John Doe",
"NotesName": "CN=John Doe/OU=ThePitt/O=GIJoe",
"AllNames": [
"CN=John Doe/OU=ThePitt/O=GIJoe",
"John Doe/ThePitt/GIJoe",
"John Doe",
"eMail": "",
"MailDomain": "SACMEA",
"MailServer": "CN=PittServer42/OU=ThePitt/O=GIJoe",
"MailFile": "mail/jdoe.NSF",
"Empnum": "0815",
"Empcc": "4711"

The form makes extensive use of @NameLookup and looks in DXL quite simple.


Building a shared approval frontend in XPages

The saying goes: "God would not have been able to create the world in 7 days, if there would have been an installed base to take care of". As much as we wish to have divine powers, we need to make with less and look after an installed base. Point in case: You have a set of approval applications in classic Notes client code, like: Travel, Leave, Expenses, Gifts, Training, BoM changes etc. You are challenged to provide a web and/or mobile interface for them.
Considering your options you have to decide between options:
  1. Dump the applications and rebuild them on a new platform. Hope that it will take long enough, so nobody will ask you to migrate any of the existing data
  2. Enable them one-by-one
  3. Hire Redpill to do an asymmetric modernization
  4. Build a lateral approval screen across your applications and leave the full web enablement for later
While #1 would allow you to silently dump all your technical debt, it will take too long and a potential data migration monster would lurk in the dark. #2 is what non-techies would expect, but again you spend too much time and you will carry forward new technical debt. After all the applications are more similar than different. Hiring Redpill would be a sensible option, but that requires board approval or is against your "we-do-it-inhouse" policy or you simply want to expand your skill horizon.
This leaves you with #4. To do this successful, you need to define your scope clearly. In the rest of the article I will work with the following assumptions, if they don't fit your situation, adjust your plan:
  • The goal is to provide the decision making screen only, not a complete workflow engine (it could evolve to that)
  • Request submission happens in the existing application, providing a new interface for that is outside the current scope
  • The flow logic (how many approvers per level etc.) is determined on submission by the original application
  • Someone from a different department indicated interest to use the approval screen from a complete different system (that is the "scope creep" that happens in any project)
When you look at workflow systems, while planning your application, you will notice, that there are 2 types of data: the flow state/sequence and the specific request data. The flow state is stuff like: what's the current status, what are the potential decisions (usually: Yes, No, More info please), who are the past, current, future approvers. The specific request data is stuff like: duration and type of leave, price of items to purchase, customer involved etc.
In your existing applications the two are stored together (and will stay like that), but you need to extract the flow data to your new approvalCentral application. On the back of a napkin you draw this sequence:
Your approval workflow
(Image created with JS Sequence Diagrams and postprocessed with InkScape)

The green part of the application exists, the first question is: how to design the submission. There are a number of options you could choose from:
  • Have the Approval Central application poll participating applications on schedule. That option is least efficient, but might be your only choice if you can't touch the existing apps at all (happens more often than you think, when admins are paranoid policies are set very cautious
  • Have each application create a flow document directly in the workflow central application. You could use a inherited LotusScript library for that. While that might have been the prefered choice a decade ago, it has two problems: first you limit your participating applications to Notes only (and there are other applications with decision requirements) and you would expose the inner workings of your application prematurely
  • Use a web service interface to implement Contract first development. While REST is the current IT fashion, you pick SOAP. The simple reason: LotusScript can talk SOAP quite well via the Webservice consumer design element, while REST would require your notification code being written in Java. So a SOAP service it is


{{Mustache}} Helper for Domino and XPages

Previously we had a look, how to use Mustache with the CKEditor to have an editing environment for templates. I glossed over the part where to store and how to use these templates. Let me continue there.
I'll store the template in Notes documents and use an application managed bean to transform a document (and later other things) using these templates.
I have 2 use cases in mind: one is to allow the configuration of HTML notification messages in applications and the other to configure the display of workflow details (more on Workflow in a later post). The approach has two parts:
  1. loading (or reloading) the existing templates
  2. using the templates
The first one should need to run only once in the application lifecycle. Since managed beans can't have constructors with parameters, I created a seperate load function, so inside the bean there is nothing that depends on the XPages runtime environment. You therefore can use that bean in an agent, application or a plug-in.
I also opted to load all templates into memory at once, but only compile them when actually needed. If you plan to have lots of them, you want to look for a different solution.
The class that does the work is the To make it easy to use, I wrap it into a managed bean and a small SSJS Helper function:
<?xml version="1.0" encoding="UTF-8"?>
    <!--  In a production system that should be application -->
  <!--AUTOGEN-START-BUILDER: Automatically generated by IBM Domino Designer. Do not modify.-->
  <!--AUTOGEN-END-BUILDER: End of automatically generated section-->

function applyTemplate(doc, templateName) {
	try {
		// Check for init
		if (!Mustache.isInitialized()) {
			var templateView:NotesView = database.getView("templates");
			Mustache.initFromView(session, templateView);
		return Mustache.renderDocumentToString(templateName,doc);
	} catch (e) {
		return e.message;

With this done transforming a document becomes a one liner in SSJS: return applyTemplate(document1.getDocument(),"Color");. To demo how it works, I created a sample database.


Custom REST service in XPages using a service bean

Talking to your backend using JSON and REST is all the rage for contemporary development. Domino has supported, at least reading, this access for quite a while using ?ReadViewEntries[&OutputFormat=JSON]. Using Domino Access Services (DAS) this has been extended to read/write support for documents as well.
However, as a result, your front-end application now needs to deal with the Domino way to present data, especially the odd use of @ in JSON keys (which e.g. jquery isn't fond of). Contemporary approaches mandate that you minimize the data you send over the wire and send data in your business structure, not in your database format. Furthermore, when sending data back, you want to validate and act on the data.
In the Extension Library there is the REST control, that you can use instead of the DAS service. It allows you to define what you want to expose as XML or JSON. There are a number of predefined service, but my favorite is the customRestService. When you use the custom service, you can write JavaScript for all events happening: doGet, doPost, doPut and doDelete, but you also can use a service bean. A service bean is not a managed bean, so you don't need to specify anything in your faces-config.xml. However it is a little special. A sample XPage could look like this:
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp=""
	<h1>This is the landing page of the orgSearch Service</h1>
	<p>Please use "search.xsp/json" for the actual query</p>

	<xe:restService id="JSONSearch" pathInfo="json" state="false">
			<xe:customRestService contentType="application/json"
if your page name is demo.xsp then the access to the service based on the pathInfo property is demo.xsp/json.


Rethinking the MimeDocument data source

Tim (we miss you) and Jesse had the idea to store beans in Mime documents, which became an OpenNTF project.
I love that idea and was musing how to make it more "domino like". In its binary format, a serialized bean can't be used for showing view data, nor can one be sure that it can be transported or deserialized other than through the same class version as the creator (this is why Serialized wants to have a serialid).
With a little extra work, that becomes actually quite easy: Enter JAXB. Serializing a bean to XML (I hear howling from the JSON camp) allows for a number of interesting options:
  • The MIME data generated in the document becomes human readable
  • If the class changes a litte de-serialization will still work, if it changes a lot it can be deserialized to an XML Document
  • Values can be extracted using XPath to write them into the MIME header and/or regular Notes items - making it accessible for use in views
  • Since XML is text, full text search will capture the content
  • Using a stylesheet a fully human readable version can be stored with the original MIME (good to eMail)
I haven't sorted out the details, but lets look at some of the building blocks. Who ever has seen me demo XPages will recognize the fruit class. The difference here: I added the XML annotations for a successful serialization:
package test;

import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement(name = "Fruit", namespace = "")
public class Fruit {
    @XmlAttribute(name = "name")
    private String  name;
    @XmlElement(name = "color")
    private String  color;
    @XmlElement(name = "taste")
    private String  taste;
    @XmlAttribute(name = "smell")
    private String  smell;
    public String getSmell() {
        return this.smell;

    public void setSmell(String smell) {
        this.smell = smell;

    public Fruit() {
        // Default constructor

    public Fruit(final String name, final String color, final String taste, final String smell) { = name;
        this.color = color;
        this.taste = taste;
        this.smell = smell;

    public final String getColor() {
        return this.color;

    public final String getName() {

    public final String getTaste() {
        return this.taste;
    public final void setColor(String color) {
        this.color = color;

    public final void setName(String name) { = name;

    public final void setTaste(String taste) {
        this.taste = taste;

The function (probably in a manager class or instance) to turn that into a Document is quite short. The serialization of JAXB would allow to directly serialize it into a Stream or String, but we need the XML Document step to be able to apply the XPath.
public org.w3c.dom.Document getDocument(Fruit fruit) throws ParserConfigurationException, JAXBException {
		DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
		DocumentBuilder db = dbf.newDocumentBuilder();
		org.w3c.dom.Document doc = db.newDocument();
		JAXBContext context = JAXBContext.newInstance(fruit.getClass());
		Marshaller m = context.createMarshaller();
		m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
		m.marshal(fruit, doc);
		return doc;

The little catch here: There is a Document in the lotus.domino package as well as in the org.w3c.dom package. You need to take care not to confuse the two. Saving it into a document including the style sheet (to make it pretty) is short too. The function provides the stylesheet as a w3c Document and the list of fields to extract as a key (the field) - value (the XPath) map. Something like this:


Domino Design Pattern: Secret documents

Domino's stronghold is security. However security is only as good as you design it. A frequent requirement in applications is to store a data set that is partially confidential and partially available for a wider audience. When you store these 2 data sets in one document, it isn't too hard to have the confidential information slip out:
  • using the document properties in a Notes client
  • using the document rest service
  • the property control from openNTF
In a nutshell: if you have 2 sets of data with different levels of read access requirements, don't store them in one document. A well working pattern in Domino is the "Secret Document". The following picture illustrates the concept:
Use 2 documents to store 2 sets of information security requirements
The user is presented with one form, but saving the entered data is done in two documents. The documents are cross referenced using the UNID. This can happen two way (as shown in the picture): the public document's UNID is saved in the secret document and vice versa - or - one way, with only the secret ID in the public document. A few pointers:
  • Based on the application's need some of the public data get repeated inside the secret document if that needs to be displayed on its own (e.g. a salary list in an HR application)
  • To avoid data drifting apart the respective data would only get updated in the public document ever and then copied to the secret document. In classic Notes that is done using a on-change agent, while in XPages a session-as-signer code snippet will suffice.
  • For very sensitive data (like even the normal user shall not see), these data sets could be stored in their own encrypted NSF. Then the UNID might not be enough, but the full notes:// url would make more sense
  • In classic Notes the embedded form editor makes the user experience with 2 documents seamless
  • In XPages two (or more) data sources sitting on one page will do the trick
As usual YMMV


Mustache and CKEditor - Round two

Having just a few static values in the CK Editor drop down list really doesn't cut it. So we extend the bean today to have more flexible options. There are a few that spring to mind:
  1. List of all items in a given document
  2. List of all fields in a form (including subforms), eventually with or without the $ fields
  3. List of provided field names
So here we go:
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="">
	<xp:scriptBlock id="scriptBlock1">
		<xp:this.value><![CDATA[#{javascript:mustache.getFormFields(database, "Memo,Person", false);}]]></xp:this.value>
	<h1>Mustache and CKEdit demo</h1>
	<xp:inputRichText id="inputRichText1">
			<xp:dojoAttribute name="extraPlugins" value="mustache">

The big change here is the replacement of the EL Expression mustache.sampleData with a SSJS expression, so we can hand over all needed parameters. The pattern is the same for the other variations, so I won't repeat it further onwards. The interesting part is the Java method. Since a form might contain subforms you are interested it, I use an array like sting, that I'll split and a boolean parameter to include system fields. Of course one could vary the approach and automatically figure out the subforms in use (have fun with that once it is conditionally computed) or first present a list of forms and then the respective fields. Also, not to depend on some magic, I add the database dependency as parameter. So you have options to play with.
    public String getFormFields(Database db, String formNameString, boolean includeSystemFields) {
		StringBuilder result = new StringBuilder();
		// Get a sorted set first
		Set fieldNames = new TreeSet();
		String[] formNames = formNameString.split(",");
		for (String formName : formNames) {
            try {
	            Form form = db.getForm(formName);
	            if (form != null) {
	            	Vector fields = form.getFields();
	            	for (int i = 0; i < fields.size(); i++) {
	            		String curField = fields.get(i).toString();
	            		if (includeSystemFields || !curField.startsWith("$")) {
            } catch (NotesException e) {
	           // Too bad
		// Now the content
		for (String f : fieldNames) {
			this.add(result, f);
		return result.toString();


CKEditor and Mustache become friends

In the beginning there was WordStar and CSV. The possibility of (then printed) personalized mass-communication had arrived in the form of mail-merge. For Notes eMails that is still a challenge (the latest version of OpenOffice now seems to have a reasonable eMail-Merge, but that's off topic here) since creating the template message with variables to be filled is kind of fuzzy (a.k.a usually out of the reach of mere mortal users).
XPages, Mustache and CKEditor to the rescue! The CKEditor shipping with XPages can be easily customized, so adding a dropdown that inserts Mustache markup shouldn't be too hard. To allow easy reuse of the code, I created a bean than contains all needed components. Add it to a page and simply call the bean functions. The first sample, using only uses static therefore is quite simple:

<xp:view xmlns:xp="">
	<xp:scriptBlock id="scriptBlock1" value="#{mustache.sampleData}">
	<h1>Mustache and CKEdit demo</h1>
	<xp:inputRichText id="inputRichText1">
			<xp:dojoAttribute name="extraPlugins" value="mustache">

The variable mustache is configured as a managed bean in faces-config.xml:

While it seems a little overkill to move the JavaScript into a Java function, we will see that the bean, once completed, is quite useful and keeps what we need in one place.


Domino Development - Back to Basics - Part 7: Map Reduce Domino Style

One of the odd things about Domino is the way things are called. It is Memo instead of eMail, Replication instead of Sync, Note store/Document store instead of NoSQL etc. The simple reason is the fact, that all these capabilities predate the more common terms and there was no label for them when Notes had them. In NoSQL circles MapReduce is a hot topic. Introduced by Google, now part of Apache Hadoop it can be found in MongoDB, Apache CouchDB and others. Interestingly it seems that short of Hadoop the mapping doesn't run distributed, but on a single server.
So what about Domino? Holding up the tradition of odd naming, the equivalent of MapReduce is Categorized View where View is the Map component and Categorize is the Reduce capability. The mapping part gets accomplished using the venerable @Formula (pronounced AT-formula) language which got inspired by LISP. If you ever wondered why you had to sort out all "yellow triangles" in kindergarden (a.k.a set theory), once you get started with @Formulas, that knowledge comes in handy. You mainly apply transformations using a set of @functions to lists of values (even a single value is a list, a list with one member). While there is a Loop construct, staying with sets/lists is highly efficient.
In its simplest case a column in a view is simply the name of the Notes item in a document (which will return an empty value if no such item exists). The formulas are used in 2 places: defining the column values of the data that is returned and selecting the documents to be included in the composition of the view. The default selection is SELECT @All which lists all documents in that database regardless of form used (if any) to create them or items contained in them. This is very different from tables/views in an RDBMS where each line item has the same content. Typically Notes application designers make sure, that all documents share a common set of items, so when combined in a view something useful can be shown. Often these item names are lifted from the mail template:Subject, Categories, From, Body.
Sorting resulting values is done as a property of the column (second tab in the property box), where sorting is from left to right. In classic Domino you can find views where specific columns are listed twice: once in the position where the user wants to see them, secondly in the sequence needed to sort, but with the attribute "hide this column". In XPages this is no longer necessary, since the view control allows to position columns in a different sequence than the underlying view.
When you categorize a column, typically starting at the first column, Domino offers a set of functions for the other columns to compute reduce values:
Reduce in Domino
You can access the values using ?OpenView&CollapseAll in Domino's REST API or in code using a ViewNavigator. A categorized view always starts with a category, so you begin with a .getFirst() followed by .GetNextSibling() for one level categories or .GetNextCategory() if you have multiple levels of categories. This capability helps when you aggregate data for graphics or pivot tables or [insert your idea here]


Domino Development - Back to Basics - Part 6: Better safe than sorry - Security

Continuing from Part 5, this installment will shed a light on security.
Domino applications are protected by a hirarchical control system. If you fail to pass one hierachy level's test, it doesn't matter if a lower level would be compatible to you current credentials. E.g. when a database would allow anonymous access, but the server is configured to require authentication, you must authenticate. To fully understand the options, sit back and recall the difference between authentication and authorization. The former establishes who you are, the later what you can do. Let's look at both parts:


When using a Notes client (including XPiNC) or running a Domino server, the identity is established using a public-private key challenge. The key is stored in the file and (usually) protected by a password. So it qualifies as a 2 factor authentication (you need something, you know something). The beauty of this PKI approach is the availability of keys for signatures and encryption. Since users hate password entries you can perform unlocking of the using various methods for single signon.
Accessing Domino through http(s) (or SMTP, IMAP and POP) uses open standards based authentication. I will focus on the http(s) options here. When a resource is protected (more on that later) the user is presented with an authentication challenge. In its simplest form it is http BASIC authentication that prompts for a username and password. In web UIs this isn't en vogue anymore, but in REST based access pulling and pushing JSON or XML it is still quite popular, since it is simple to use.
The most prevalent form (pun intended) is form based authentication. If a protected resource requires higher access than the current known user (who is anonymous before logging in), the user is redirected to a login page, where username and password are requested. This authentication can be per server or for a group of servers using LTPA or 3rd party plug-ins.
Users typically are listed in the Domino Directory or any compatible LDAP directory configured as authentication source. Users on Domino mail need to be in the Domino directory for mail routing to work.
A typical (structural) mistake: trying to use a remote 3rd party LDAP which creates additional network latency and single point of failure (the remote LDAP). When your Domino application server is running, your directory service is available, I wouldn't step away from this robustness. If your users strategically are maintained in a more fragile directory, use TDI to keep them in sync.
The final option to authenticate users is the use of X509 certificates. This requires the public X509 key to reside in the directory and the private key in the user's browser. I haven't seen that in large rollouts since it is a pain to administrate.
Anyway, as a result of an authentication, a user is identified with an X500 compatible name:
CN=Firstname LastName/OU=OrgUnit4/OU=OrgUnit3/OU=OrgUnit2/OU=OrgUnit1/O=Organisation/C=Country
Country (/C) and the Organisation Units (/OU) are optional. Typically we see 1-2 OrgUnits in use. They are practical to distinguish permissions and profiles as well as a remedy for duplicate names. If not taken care of carefully, users authenticated through a 3rd party LDAP will follow LDAP naming conventions, which is a PITA. When @UserName() doesn't start with CN= (unless it returns Anonymous) you need to take your admins to task.
In any case, it is transparent for an application developer on Domino how a user authenticates, you only need to care that it is happening when you protect a resource. Keep in mind: authentication is only secure via https!


Now you know who the user is, you define what (s)he can do. Domino uses 2 mechanism: the Access Control List (ACL) to define read/write access levels and the Execution Control List (ECL) to define what code a user can run. The first level of authorization is server access:

It doesn't matter what access a user might have inside a database, if the server won't let the user access its resources. So having your admin locking down the server properly is your first line of defense. Server access is defined in the server document that also contains the ECL for the server. A normal user doesn't need ECL permissions, only the ID that is associated with the XPage (or agent) that a user wants to run. "Associated" here means: the ID the code was signed with. By default every time a developer saves an XPage, that page gets signed with the developer's ID. This is very different from file based application servers where a file doesn't contain a signature, but similar to the JAR signing in the Java world. Keep in mind: in Java that is a manual process using a command line tool and requires the acquisition of an (expensive) code signing certificate, while in Domino it is automatic.
Common practise however is, not to allow a developer to run code on a production server (unless you are Paul), but signing the database with either the or a specific signer ID. This step would be performed by your admin.
When you access an NSF, the access level is set in the ACL to one of these values: No Access, Depositor, Reader, Author, Editor, Designer, Manager.
Additionally a user can have roles assigned to her, that allow a refinement of interaction and access. Roles are always used with the name in square brackets. Typical roles are [server] or [Admin]. Roles are defined per NSF. The following tables shows the permitted operations per access level:

Read Access

Having read access to a database doesn't automatically make all documents visible. They still can be protected by reader fields. More below

Write Access

Having Author access to the database does enable a user to edit any document where the user is listed in one or more items of type author, While Editor access allows to edit any document the user can see. So as a rule of thumb:
The access level for normal users to a Notes/Domino based application should be set to Author

Reader & Author fields

They are quite unique to Domino, since they allow declarative protection of a note. If you implement (in code) something similar in an RDBMS, you get quite a complex relation:

Looking at the above table you could conclude: Author fields are only relevant for users with exactly Author access to a database: if the access is lower they won't be able to edit anything, if the access is higher, they can edit anything. However in conjunction with Reader fields they have a second purpose. Reader protection is defined as:
When a document has at least one item of type Readers that has content (non-empty), then access to that document is restricted to users and servers that are explicitly (by name) or implicitly (by group membership, role or wildcard) listed in any of the Reader or Author items.
A typical pattern in a Notes application would be a computed Author field with the value [server]. The application server would get the role in the ACL. Since a server most likely has Manager access anyhow, it has initially no impact on the application. However when some mechanism activates reader protection, this field ensures that the server can see all documents. It is important to use an Author field/item here, to avoid triggering read protection where it isn't necessary. Reader fields have a performance cost, so use them wisely.
A popular mechanism is to provide a checkbox "Confidential" and then compute a reader field: @if(Confidential!="";DocumentPeople;""), where "DocumentPeople" would be an item of type Names. In workflow applications, the content of reader and author fields isn't static, but changes with the progress of the workflow. Careful planning is required.

There is more

Notes can refine access further. Documents can be encrypted and signed, as well as sections in a document can be access controlled or signed. However those capabilities are not yet exposed to the XPages API and are thus of limited interest for new XPages developers.

Further readings

There are a number of related articles I wrote over the years: Questions? Leave a comment or ask on Stackoverflow


Domino Development - Back to Basics - Part 5: Finding data - Collections and Search

Continuing from Part 4, this installment will clarify finding data. It, again, is different from other database concepts. Read on:
In a RDBMS your data retrieval queries are formulated in SQL, while XML, RDF or Graph databases might use sparQL, Gremlin, Cypher or JavaScript. There is no shortage of Query languages (one you should master in any case is XPath, get the full reference from the inventor himself).
On a first look a Query Language seems absent from IBM Notes. On a second look, there are two.

The IBM Notes Query Languages

Access to Notes data mostly happens using a view, so the pre-selection happens using the SELECT statement and the @Formula language. We will see how to narrow results down, just below. But you are not limited to creating a view first, a database can be queried directly using @Formula when executing"some @Formula expression"). This works like any other query language around. This flexibility comes with a price: speed. is the slowest way to retrieve data from a NSF and you shouldn't use that for your default access path (that is: the actions most users will use in a regular manner).
The second query language is the composition of a fulltext query. In its simplest form it is just some text you want to find, but it can be extended to specify the fields to look into and construct additional constraints. It takes a little to get used to it, but it is quite powerful, just one example: fish SENTENCE chips finds documents where fish & chips are mentioned in the same sentence. Important note here: make sure the FullText index is configured, so it isn't crawling slow.

Finding data

There are several ways data retrieval happens in IBM Notes:
  1. Direct access
    Notes documents can be addressed using a notes:// address. The session class has a resolve(notesurl) method (not the fastest) and the database class has Database.getDocumentByUNID(id). Don't confuse it with Database.getDocumentByURL(...) which created a document based on HTML content at the specified URL. Internally the byUNID is used whenever you click on a Notes document link. It is one of the fastest access methods. There is also byID, which you should avoid, since it isn't save across replicas or copy style compaction
  2. View/Folder access
    The view selection formula (for folders: action taked by users or code) defines a set of documents shown in a view. Using one of the collections (see below) you can process all those using View.getAllEntries, View.createViewNav() or by directly starting to loop through the documents using View.getFirstDocument() (or getFirstEntry()). When looping through them, any saving back of a document might remove it from the view index, so you need to fetch the next document/entry before that
  3. Subsets of View/Folder
    This is the most typical query access to Notes data. To make this work the view needs to be sorted or categorized (depending on the method). A typical example in a workflow application would be a view with the selection formula SELECT Status="Pending Approval" and the first column categorized with CurrentApprover. You would retrieve documents using View.getAllDocumentsByKey(...). This method works already with sorted columns (don't need to be categorized) and can take a Vector as search term. It would then match the Vector elements to the sorted columns for search.
    However using categorization has the advantage, that you can directly address that view in an URL using ?OpenView&RestrictToCategory=search term and the view control has a matching property as well. An alternate approach is to use View.getDocumentByKey(...) to jump to the first Document/Entry and then loop until the desired end. This approach is useful when you might not process all entries or process beyond the current key (e.g. process all requests from R-Z)
  4. FullText search
    Make sure you actually have a FTIndex on the database, otherwise that will be slow. The syntax takes a while to get used to. There is one FTIndex per database, even while you can limit the search to one view
  5. Database.Search(...)
    The most flexible search, with the price of speed and sort order: slow and you have to sort it yourself. Don't use unless unavoidable
Depending on your needs pick one or the other


Depending on your access route choosen above, you might get one of the 4 result collections. Those are no proper Java Collections, but come in their own little API implementation, unless you use a well designed Notes API. You will find getFirst and getNext(currentOne) methods
  • DocumentCollection
    The oldest and most flexible collection. Can originate from a database, a view, a search. Since you have full access to all documents in it, it is the slowest collection. It was also the first collection available in the API and is a good candidate for being replaced by a ViewEntryCollection when tuning your database. A DocumentCollection contains only data documents. Is has fancy set operations
  • NoteCollection
    The NoteCollection can contain data and design Note elements and is slightly faster to build than the DocumentCollection, since it initializes a Note class only on access. So the total time "creation + access" is similar to the DocumentCollection. The main use case is access to design elements and some fancy set operations
  • ViewEntryCollection
    This collection provides all (or selected) documents from a view. Instead of opening each document it gives access to the column values in the ViewEntry object, as well as the same fancy set operations you learned to love from the DocumentCollection and the NoteCollection. The ViewEntryCollection performs faster, given you have all values you need in a view column
  • ViewNavigator
    Technically it isn't a collection, but the ability to point around in the view index. So you won't get any set operation to refine the result, but you get speed. You can jump from category to category (great for reading the reduce() values) and run through an entire view very very quickly. The ViewNavigator is a main reason to plan your views well

Next up: Better save than sorry - Security


Domino Development - Back to Basics - Part 4: Domino views are different

Continuing from Part 3, this part is typically the hardest to understand when coming from an RDBMS background. So take your time.

Domino Views are different

In Domino data is typically accessed via a view, but views are different than the ones you know in an RDBMS. The following table should provide an rough overview.
Data Defined in a database schema. Data is contained in tables. All records in a table are uniformly the same. Fields without values are there but empty. Different types of data require different tables. New data requires ALTER TABLE statements wich affects existing data (and often requires down time). Each column can have one value Data is contained in documents. Documents are schema free (there is a Meta Schema). Items in documents (the closest to a column in a table) can have multiple values. Each document can have a different set of items. New data is added as needed, no downtime or change of existing documents required
Views Selection of rows, often accompanied with a JOIN operation to denormalize normalized data. Can be all data of a table or a subset, picked by a WHERE part in the SELECT statement.
By definition views don't contain any data, but pull them ad hoc from the participating tables.
Various RDBMS systems use indexes on keys used in SQL statements to improve performance. The ad-hoc nature of the queries offers maximum flexibility at CPU and I/O cost.
There is a whole industry around SQL Query Optimization (and there is the whole story about SQL Injection attacks even by mothers)
  • are pre-created and updated by a view indexer task when documents are created or updated.
  • They do occupy space inside an NSF (there are discussions in development to externalize them to reduce backup load and allow more I/O tuning).
  • Since they don't require a document to be opened, they are fast (unless you use performance killers - see below)
  • Each column in a view shows data. Using Domino designer that data can be different for each row (defined by @Formula
  • Since items can have multiple values, they can appear more than once in a view (controlled by the view design)
  • Different document types (e.g. a customer entry and an order entry) can show up in the same view without the need for a JOIN operation (which anyway isn't a feature of a Notes view - this is one more reason why Notes favours inheritance over relation)
  • Accessing a view using a ViewNavigator is very fast
  • A view can contain data that is computed and not contained in the documents. Typical uses are translations of status codes to human readable text)
Hierarchy SQL Tables and views are flat by definition. Some SQL extensions (e.g. Oracle) allow hierarchical queries, but the result will be a flat query result.
  • views can be flat or hierarchical
  • A common use for hierachical views is the option to categorize a column value. This operation will sort the column and show each value once only above the result rows (in the UI typically with a twistie icon)
  • A categorized column also can perform some additional simple Reduce operations (albeit not distributed)
  • Programmatic categorized views are typically used with @DBLookup; @DBColumn; getAll[Documents|Entries]byKey
  • Multiple columns can be categorized
  • Typically the categorized columns are the first ones on the left, but that isn't necessary
  • Subcategories can be created by having the item value containing a backslash. If an item contains more than one value, the document will appear in multiple categories. An alternative approach turns multiple values into subcategories: @Implode(fieldname;"\\")
  • Besides categorization there is a secondary hierarchy mechanism available: the response hierarchy. Defined by a view property, response documents (of all levels) are shown automatically below their respective main document. A special column can be defined that is shown only for response type documents (containing a valid $REF item). In this case only the columns left of this are shared with main documents, columns right of it are ignored for responses and only the response column is shown (check the discussion template for how it looks)
Sorting Sorting happens using an ORDER BY clause. For resorting a new select clause needs to be issued. RDBMS systems allow indexes to be created to improve sorting speed.
  • Sorting is a property of a view column. It will increase the view index size and thus I/O and reindexing time
  • Sorting happens from left to right
  • Sorted columns don't need to be visible in the view (there is a property to hide columns conditionally or in any case)
  • Columns can be resorted in the UI when the respective property is set. The custom sorting sequence is also available as URL parameter
  • A common design mistake (creating index data unused): make every column sorted, even if the sorting has no more effect on the result sequence and/or make every column resortable even if users might not use it. For the later case there is a remedy: set a column property to defer the creation of the alternate index until it is requested the first time
Definition Views are defined in SQL using any editor of your choosing and then uploaded/executed in the database. Database admin rights are required Views are defined in Domino Designer or even the standard Notes client. Users can be allowed to create "private" views only visible to themselves, or at access level Editor views that can be shared. Columns are defined by picking item names from a list, predefined functions or writing @Formula a LISP like language (easy to learn, quite powerful to use). Brave souls have been spotted using DXL to write view definitions in XML and upload them into designer (but you don't want to do that)
Selection View entries are selected by WHERE in SQL
  • The selection statement also start with SELECT, but uses the @Formula language for picking the documents
  • All documents (typically in a view named ($All)) are selected by SELECT @All
  • More samples: all customer documents by SELECT Form="Customer", incomplete workflows by SELECT @IsMember(Status;"Submitted":"In Progress")
  • Overly complex select statements slow down view computation and you might need to employ some brute force view tuning
  • A performance tip is to compute the criterion for selection inside the document to result in a field value @True|@False and then limit the selection formula to SELECT showInView_Pending
Performance killers There is nothing better to make your hardware seller more happy than code that violates performance considerations. In SQL the typical performance killer are OUTER JOINS especially over many large tables. Missing indexes are another cause Since Notes views contain actual data, they need to be treated a little different. There are a few killers around:
  1. Performance killer number one is the use of time based view selection formula: @Now; @Today; @Yesterday; @Tomorrow. In one word DON'T. IBM isn't so strict, but I am. You do have options
  2. The second killer is the use of @IsResponseDoc as part of a view selection when you end up with only a few parent documents, show documents in response hierarchy and many "hidden" responses since the parent isn't displayed. Better use @AllDecendants, check the explanation for more details
  3. As mentioned above: be stingy with sorting of columns, both presorted and optional. Make sure if you define optional sorting, that you defer creation until first use
  4. Overly complex view selection formulas and too many views slow down the view indexing process, so take only as much as you really need. In classic Notes we often find countless variations of views for display. In XPages you can often combine them (given they differ only in columns) and use the XPage for the variation in look and feel
  5. Last not least: your code. Let the view indexer do its job. If you use view.refresh() everywhere "as a precaution" you pay dearly with sucking performance. A view gets locked when reindexed, so 100 users all executing a view.refresh() lock the view 100 times sequentially
Watch your SQL

Next up: Finding data - Collections and Search


Domino Development - Back to Basics - Part 3: Not all Documents are created equally

Continuing from Part 2 this installment will introduce you to the four types of documents you can find inside an NSF:
  1. Documents
  2. Responses
  3. Response to Responses
  4. Profiles

Profile Documents

All document types follow the same rules as outlined in Part 2, but differ in the way they are created or accessed. Short of profiles all documents typically get accessed in one or more views, by users or your code. Profiles on the other hand, never show up in a view, they are only accessed using some code. The easiest case is @Commmand([EditProfile]"NameOfAForm") for a System profile and @Commmand([EditProfile]"NameOfAForm";@UserName) for a user profile. Notes keeps an internal index for profiles and they are typically cached in memory, so accessing them is superfast (and quite a headache if you fall into the trap to use them for frequently updating information). Their intended use is storing configuration and selection options, so @GetProfileField(FormName;FieldName) has rapid access to options. In XPages applications the need for profiles is diminished, since you typically would store such configuration options in the Session and Application Scope where you load them once.

Document hierarchies

The Document Hierarchy
The different "Document types" are actually "only" form properties (One can see how forms and documents are points of confusion). Inside a document the presence of the $REF field (containing the UNID of the parent) makes a document a response. If the parent document happens to be a response itself, the document then is considered a "Response to Response" All Notes APIs have methods to create or query a Parent - Child hierarchy. A good overview can be found in this Ytria TechLab. To my knowledge a Response-to-response document doesn't contain a direct reference to the root document
The form properties are used when you create a document in the Notes client to automatically establish Parent-Response relationships. A typical mechanism used in response document forms is to enable inheritance (that feature can also be used for standard documents, it isn't limited to responses!): When the document get created in the context of the parent (think: you read an entry in a discussion and press reply) all items from the parent document, that have fields in the response form with the same name as the item get copied into the response document (good fun if item and field don't have the same data type). Additionally all formulas computation in the response get executed against the parent, but stored in the response. This happens once at creation, not thereafter. To keep fields in sync you would use @GetDocField($REF;"SomeFieldName") to obtain the latest value (there are methods in LotusScript, JavaScript and Java too).
Sounds complicated? Go and create a discussion database (the template is on your Notes client) and play with it for a while. All this leads to an important Notes concept:

Inheritance over Normalization

Database normalization is a corner stone concept of RDBMS. If RDBMS is all you ever used and you are stuck in the idea that databases = RDBMS (really?), this will be hard to swallow.
Normalization is an abstraction that is suitable and needed for an RDBMS, it isn't a real life fact or an absolute necessity. If you blindly apply Normalization to XML Database, document databases, Graph databases you create more problems than solutions. So free your mind and follow:
Let's look at a typical normalization example: an invoice: You have one customer table, one invoice table, one items table and one products table. In your invoice table you point to the customer table, in the item table to your invoice table and your product table. Life seems good.
Invoice Entity relationship diagram
But then pesky reality kicks in: Customers move, but legal demands that existing invoices don't change (commonly changing a legal document is called forgery). Product prices change, or sales negotiated a different price. Customers might call your articles different from your product table etc. So you adjust your diagram:
Enhanced Invoice Entity relationship diagram
The attributes in red are inherited from the respective tables. This inheritance in an RDBMS is entirely done in code, in (classic) Notes it could be done through a form property. So inheritance is also quite common in RDBMS based applications too.
In Notes it is rather the rule than the exception. Retrofitting an RDBMS normalization onto a Notes application might not bring the benefits you might expect, so keep in mind:
Notes favours inheritance over normalization.
Interestingly when you want to export data, e.g. for data interchange you hardly generate one table/file per RDBMS table, but you de-normalize the data for transport. In Notes this tandem of normalization/de-normalization isn't necessary since you deal with a (self contained) document already.

Items have multi-values and can be grouped

A typical way in a Notes application to handle parent-child records is the use of a group of related multi-value fields (in the form), resulting in a set of items in the document with the same number of values. You might have one address field (which can have many lines - much more flexible than a text column in an RDBMS) and a set of multi-value fields like: ProductID, description, Price, Quantity and SumPrice. Since the (classic) UI didn't make it very easy to keep those in sync, a module called Table Walker had been made available on the Sandbox. I later created Tablewalker for XPages as part of the XPages tutorial (with enlightened bug fixes courtesy Matt White). Keeping the line items inside a document makes transporting, processing or securing them much easier, however it makes querying the line items a bit more difficult (more on that when looking at views). You can take a similar (actually more powerful from a modelling point of view) approach when using DB/2 PureXML. DB/2 PureXML and XPages are actually a match made in heaven (or was that the other place starting with h?), which is the subject to another story, another time.
Remember (before you start bitching about parent-child relations): the explanations above are guidelines, there are valid cases where RDBMS like separation of properties and detail rows into separate documents is a better approach. You know how to normalize, so I don't need to explain that part

A Document can save directly to an RDBMS

There was an ill-fated attempt to use DB/2 as storage destination for NSF data called NSFDB2 which unfortunately predated PureXML capabilities (otherwise the story might have been different), and it shall not be spoken about it anymore (support for it has ceased anyway). The tools here, available since R5 are DECS (Domino Enterprise Connection Service, included on every Domino server and LEI (LotusIBM Enterprise Integrator, an IBM product you need to buy). Filling in some simple forms in the configuration database can link your Notes forms (and partially views) to backend data from various databases or ERP systems. Once you got a grasp of these tools, some creative usage patterns become possible (more details here about Domino and RDBMS). Of course you can use code (ODBC, JDBC or LCLSX) your data transfer or use the IBM Swiss Data Knife for moving data around.

Next up: Domino views are different


Domino Development - Back to Basics - Part 2: Forms and Documents

Continuing from Part 1 you now know that everything in Notes is stored in a Note. To further understand how Notes "ticks" some light needs to be shed on the relation between forms and documents.
In the simplest way to look at it, a form acts like the schema to be used to populate a document with values. The fields one fills into a form get stored as items into a document. This explanation is good enough for starters, but warrants a closer look to fully appreciate Notes' flexibility. In a RDBMS a record is "unbreakable" linked to the containing table by the database schema definition. Each record has each column defined in the schema. Notes ticks differently. The link between a form and a document is established by a Notes item with the reserved name "Form". When you open a form, fill it in and save it, automatically a item with the name of the form gets created. When that document gets opened, it will pull that form for display.
That link is not static: it can be overwritten in many ways:
  • In the Notes client using View - Switch Form ...
  • Though a field with the name form containing a formula
  • Code in an event
  • Code in an agent
  • A form formula in a view
  • A Smart button
A form ultimately only decides what gets displayed or computed when you open a document. Adding or removing a field from a form doesn't add or remove an item from a document! A popular trap: You add a new field to a form (e.g. color) and give that field a default value (e.g. green). You open that document to check and it nicely shows "green". However the document actually doesn't contain the item. Since the item wasn't there when you open it, the default value executes, but until you switch to edit mode and hit save nothing gets saved back into the document.
The same applies when you remove a field from a document: it simply isn't shown anymore, but the data is still in the document. If you want to get rid of the data too, you need to create an agent for that (which is pretty simple: FIELD obsoleteField := @DeleteField (You get the irony here: despite deleting an item the formula is @Deletefield. Renaming a field is the same as deleting and recreating it with a new name. It is an action in the form, with no impact on the document.
Another stumbling block: the form defines how an item is displayed and entered into a form via the field definition, suggesting data type and if it is one or more values. However items are always multi-value data and the data type of an item doesn't need to match the type of the field (if auto-conversion fails you will get an error). There is a detailed explanation about forms, documents and items you want to read.

Forms might include subforms, which from a document perspective doesn't matter. An item created (through a form) doesn't carry any information how it was created. If you need a detailed track record what item has been created/updated when and by whom, you want to have a look at the Inversion of Logging pattern.
Forms also contain a series of events that fire when things happen in the form. In the context of XPages that is no longer relevant, the event model is superseded by the XPages event model. However one area might be relevant: The @Formula used for fields:
How fields are computed
* Keywords: check boxes, radio buttons, drop down list etc. Depending on type are inherently single or multi-value or both
Fields in forms can be computed, carrying one @Formula or Input fields, carrying 0-3 @Formula. A field defined as computed for display doesn't get stored back into the document. A field computed when composed gets computed once when the document doesn't contain the field yet. Usually that applies to new document, but the formula also would run when a field is added to a form and a document is opened with that form (and if the document isn't saved, the result of that formula doesn't get stored back). A computed field gets refreshed every time a document is recomputed, that implies the document is in edit mode (unless there is no item with that name, then it runs in view mode too)!
The translation formula replaces the user input with its result. A popular prank is to pick a formula that computes something independent from the input. For orderly developers there is @ThisValue to use the user input. The Validation formula needs to return @success or @Failure("message"). The failure message gets propagated to the XPages error system and shows in the form error control, but not in a field error control.
In XPages a developer can define for each document data source that the document should be computed on open and/or save. The difference to classic form use: in XPages on open all formulas run: default, translation and validation while in classic on open only default runs. Also the form computation runs after the XPages validations, so if an XPages validation fails, the form computation never runs which then might yield additional errors from the validation formula.
Next up: Not all Documents are created equally


Domino Development - Back to Basics - Part 1: The NSF

Over the last five years I have trained many developers on XPages. A good portion of them came from a non-Notes development background: Java, dotNet, PHP and others. Most of them had a better grasp on the necessities of web development than most of their die-hard-is-there-anything-other-than-the-Notes-client LotusScript colleagues. However they struggled to fully appreciate the finer points of the NSF based nature of Domino development. The little article series is for them.
In short: Notes is a distributed application platform with an integrated NoSQL document database featuring a rich security and event model. It can be accessed in many different languages and protocols. While it is written in C, the most prominent languages used to develop applications in Notes are LotusScript (a Basic dialect, now de emphasised), JavaScript and Java. The most used protocols are HTTP(s), NRPC and SMTP.

In the beginning was the NSF

Almost everything in Notes is stored in NSF databases (NSF = Notes Storage Facility).
The core structure of a Notes database
A NSF contains properties, an Access Control List, View indexes and a collection of Note elements, hence the name Note(s). The file name is not a property stored inside the NSF, however the title and the ReplicaID are. The ReplicaID has a special meaning: two databases sharing the same ReplicaID can synchronise their content in a process called replication, typically between a client and a server or multiple servers. When a server contains two or more NSF with the same replicaID replication can happen with any of them. A typical mistake: copy a NSF on the file system "as backup" creating not a copy but a replica. There is a menu action that creates a real copy for that (File - Application - New Copy).
Inside the NSF all data is stored in Note elements: both the design of the database, as well as the actual content. A data note (a.k.a Document) can be listed in a view (with the exception of a profile document), while a design note lists only in Domino Designer (unless you hack it). Design and data note elements share the same inner structure.
NSF do not contain any schema or data definitions. It is all Note and Item.

A Note is where you find anything

An individual note contains a series of properties and a collection of items (not to confuse with a field in an RDBMS or a field in a form). Besides data values for creation and update a note carries a UniversalId, a NoteId and a serial number. The NoteId is unique inside one specific NSF, the UniversalId is the identifier to link two note elements in two NSF (that have the same ReplicaId) together, so they synchronise on replication. The UniversalId is unique inside the NSF. Two different NSF can have documents with the same UniversalId. This is not an exception, but happens e.g. on mail routing. The serial number counts the times that note has been updated and is used in conflict resolution while replicating.
the basic unit of storage is a Note
The contained items follow the same pattern: they have a name, a set of properties and a collection of values. Two important properties are name and type. The name of an item doesn't need to be unique. Quite regular, especially when looking at the type RichText you have more than one item with the same name (for RichText the unwritten convention is to use the name Body). Items that start with the $ sign are used for system properties (e.g. $HasNativeMime indicates that the note contains a MIME document rather than RichText). Items also contain a serial number for conflict resolution in replication. The values in an item share the same data type. Items are always multi-value (a Vector in Java) even if the count is just one.
Next up: Forms and Documents


Domino Development - Back to Basics - Overview

XPages development uses well established standards: JSF, JavaScript, HTML, CSS, JSON, HTTP and XML.
However it is different from other web development environments:
  • It is highly integrated. database, application and web tier come in one dense package. If your server is up Domino is good to go (no JDBC database timeout)
  • The native database is NoSQL (actually the mother-of-all-noSQL), a document oriented store
  • It is designed for distributed, partially connected operation
  • Code can run on a server or on a client
  • Domino provides build in directory and messaging capabilities
  • It provides a hierarchical, declarative security model
To ease new developers into "how Notes is different" I will publish a series of articles, that focus on the big picture. They will make it easier to avoid "coding against the grain" and to appreciate the power of Domino. The plan so far:
  1. The NSF
  2. Forms and Documents
  3. Not all Documents are created equally
  4. Domino views are different
  5. Finding data - Collections and Search
  6. Better save than sorry - Security
  7. Map Reduce Domino Style
Stay tuned


Driving Embedded Experiences Adoption

With the introduction of Connections Mail and IBM Notes 9.0 a brand new productivity feature was made available in eMail: Embedded Experiences (EE). They are defined by the Open Social Foundation and can also be found in IBM Connections or Atlassian's Confluence Wiki.
The use case for embedded experiences in eMail is simple:
"Any application that sends a notification message to your attention and/or action can reduce the time and clicks required for processing by taking advantage of an embedded experience"
Ryan described in detail how to switch on the processing of embedded experiences and the Wiki tells how how to use XPages to send them. The trouble with this: you need to touch every single application that sends out a notification. You might not have time, budget or source code access to them.
So are EE a nice idea, but confined to some niche applications?
Not at all! When peeking under the hood, you can see, that an EE enabled eMail has a HTML and a JSON Mime part. If a client doesn't understand EE (like mobile clients or any other eMail than IBM Notes), then the HTML message is rendered. Important here: even with Notes you must have a HTML part, a plain text message won't do. Using a "when new mail arrives" agent, you can turn an incoming message into an EE.
As long as the target system supports one of the authentications the Account API provides (Basic HTTP, JEE Form, SAML, Spengo, OS Credential, Siteminder, LTPA) this will work. This is Notes client only, for Connections or iNotes you are limited to SAML.
First thought was to use a "Before mail arrives" agent to make sure, that the mail gets delivered fully converted. However such an agent runs in the router context and might be a performance killer - the when new mail arrives option seems a suitable compromise. The approach works with signed eMails too, since signing covers the eMail body, but not headers and mime separators (you can't sign the header since each routing hop adds information there).
As an example I used a plain text eMail, that sends a notification with a number of field/values separated by colon and containing only one URL. For your use cases, you will need to adopt the code (or wait for a future blog, just saying). Since my original message only contained text/plain I had to add text/html and application/json mime parts.


Aggregating view data for use in d3js graphics

QuickImage Category
Dashboards are all the rage, so it is natural that your XPages application need a dash of a dashboard. A view makes an excellent source for dashboard data and the ability to categorize views handles the heavy lifting of aggregating the values you want to use e.g. in a bar or pie chart. I've been fallen in love with d3js since she is the ultimate visualization (if in doubt, read the classics).
D3Js is a harsh mistress of exceptional beauty, so you might want to check out some of her offsprings like RickShaw, NVD3 or xCharts. Indulge in myriads of tutorials and reviews (check part 2 too) and the over 1000 examples.
With just a few lines of JavaScript any categorized view can be used as source for d3js. The script can read a categorized view with one to three categories. If you have one it reads the view, for two you get hierarchical data or provide a key, so you only retrieve the selected category etc.


Extend the Replicator

QuickImage Category  
One elegant way to improve perceived performance is to run computing task outside of user time. In Notes that is done using (scheduled) agents and scheduled replication (On mobile devices it is called PushMail ). When you have longer running tasks that only make sense when new data might have arrived, a scheduled agent doesn't make much sense.
Triggering a process "On Replication" is much preferable. Classic Notes agents don't have this ability, but the Notes full client can do that. Already today the replicator hosts not only classic Notes NSF replication, but also address sync and Activities sync. This type of extensions can easily be created using an Eclipse extension point:
  1. <extension point="">
  2. <unit class="com.notessensei.demo.RunOnReplication"
  3. id="com.notessensei.demo.runonreplication"
  4. image="icons/replicate.gif"
  5. label="NotesSensei's Demo Replication">
  6. </unit>
  7. </extension>
There are a few caveats (besides designing an 32x32 icon) when building the class, but once you know them it is straight forward:
  • You either extend or
  • For both approached you must have a default constructor without parameter. Neither of classes you extend has one, but your implementation needs one
  • When subclassing Job you extend run() for NotesSessionJob you extend runInNotesThread(). The later has the advantage to have all the session handling completed, but you end with a dependency on Notes client code
  • While both methods provide an IProgressMonitor as parameter, your code needs to try to obtain the Notes specific one (see sample)
  • Keeping track of units of work is a pain, expecially when your tasks calls various modules with unknown number of steps. Luckily there is SubMonitor that allows to specify a set amount of "ticks" for a subtask that can then adjust them as needed (see example)
  • You should check IProgressMonitor.isCanceled() inside your code and terminate processing at the first possible point (without sacrificing data integrity) when encountered
Your class could look like this:


Extracting data from Domino into PDF using XSLT and XSL:FO (Part 5)

QuickImage Category  
This entry is part of the series Domino, XSL:FO and XSLT that dives into the use of XSLT and XSL:FO in IBM Lotus Domino.

XSL:FO and XSLT are text based formats, so you could use your favorite text editor (or a modern heir) to write your XML. You also can poke yourself in they eye.
A XSL* transformation is code running with pattern matching, priorities (and closing tags), so the probability you get it right, especially when you are new to the domain approaches zero. A good content aware editor and a debugger tool is paramount. When looking around, most of these editors are geared towards developers. Here are some I tested or worked with: There are more, so you want to do some research yourself. Of course you can debug the hard way or just use the Eclipse XSLT Debugger.
Being able to write good XSLT, doesn't make you write good XSL:FO automatically. So you want to make sure the tools support that as well. The above tools do FO too and there are some specialized editors around to choose from. However they have limited appeal to a non-developer user.
A suitable tool for business users (still they would need to have some IT aptitude is the Java4Less FO Designer. It is modeled not based on an XML tree, but like other report generators, so business users might feel more familiar:

The price is modest and you only need licences for people who want to design FO reports, not to run them (that's pure Java, as you learned already).
As usual YMMV.


Protecting your XPages Application

QuickImage Category
One of the hallmarks and success factors of IBM Notes was the openness of the design of all Notes applications. Since lots of people put blood, sweat and tears into their applications, Lotus added the possibility to hide the design of an application to protect it from spying eyes. While I don't like hidden designs I can understand the rationale.
Others keep their design open, but compile the LotusScript libraries with references to lss files outside, so the core business logic is protected.
Hiding the design doesn't work for XPages, you can poke around and get back to the source. So what are your options?
  • Keep your business logic in Java classes (Pojo, Beans, Managed Beans) and develop them outside of the NSF and transfer them as jar file only
  • Deliver your custom controls as Extension Library (which can be deployed using an NSF) - which obviously requires advanced Skills
  • Keep your XPages lean and handle the delicate parts in custom controls that you hide (see below)
Of course compiled Java can be transformed back to Java source code, so you might want to add another layer using Java obfuscation, but then you might have crossed the line between precaution and paranoia (to my best knowledge IBM doesn't use obfuscation in Java products). An additional level of protection is offered using Refuctoring (no, that's not a typo).
So how would a typical workflow look like? There seem to be quite some moving parts involved:
Protecting XPages Workflow
  1. Develop and test your application as per normal. Keep items you want to protect in custom controls and Java classes
  2. Make sure you have your project build before doing the next steps
  3. Link your database to an On Disk Project for version control (but you do that anyway don't you?). Let's call that project "OnDisk Open"
  4. Have your target database (the one that you will deliver to your customer) setup in a similar way. Let's call that project "OnDisk Closed"
  5. Copy all elements you want to keep open (Forms, Views, XPages, CSS etc.) from OnDisk Open to OnDisk Closed
  6. Copy your Java source files and the generated Java source files for your custom controls (found in the Local directory, actually outside the NSF) in OnDisk Open to a src structure. Custom controls live in the xsp package. Your code lives in your packages
  7. Compile your Java
  8. Put all the compiled classes into a jar
  9. Copy that jar into the /WebContent/WEB-INF/lib folder of the OnDisk Closed project. At the first time: add this jar to the classpath of your closed.nsf after sync with the OnDisk Closed project
  10. Copy the .xsp-config files of the custom controls you processed in the previous 4 steps into the /WebContent/WEB-INF/ directory (the same directory your faces-config.xml is living in)
  11. Sync the OnDisk Closed project with closed.nsf
  12. Build your closed.nsf - done
With a little skill those steps can be automated.


NotesSessions, XPages and Threads

QuickImage Category
When you build an XPages application, long running operations will time out. After all there is a timing limit how long a browser (or XPiNC) client (and the user) will wait for a response. You could resort to launching an agent via a console command, but besides the security consideration it is quite a hack and headache. Mixing XPages and agents doesn't have a happy ending (and should be confined to the upgrading phase of your classic application).
Enter Java multi-threading. I call it "Teenage-mode": your code does multiple things at the same time and doesn't seem to get distracted.
When you read the classics you will find threading hard and rather threatening (pun intended), but thanks to progress in Java6 and above it isn't too hard.
There is the Executor framework that handles all your Runnables. There are 2 challenges to overcome: one specific to Notes and one general to all concurrency programming. The former: The Notes Session class is not threadsave and can't be shared between threads, the later: you have to get your queueing right:

(Image courtesy of Lee Chee Chew)
To get a new Session object, we have the session cloner, thanks to the extension library. All your threads implement the Runnable interface that is later called by the executor framework. There are 2 approaches for background task: one where you expect the thread to return some value at some time and the other where a thread runs its course and terminates silently when done. For a current project I opted for the later. The threads contain a callback class where they report back what they have to say. To make it all work we need:
  1. A management class that executes the threads for us. Preferably long living (like session or application beans)
  2. A cloned Notes Session
  3. One or more classes implementing the runnable interface
Let's look at some code, first the class that runs in the background:
package com.notessensei.demo;

import java.util.Collection;
import java.util.Map;
import lotus.domino.Database;
import lotus.domino.NotesException;
import lotus.domino.Session;

public abstract class AbstractNotesBackgroundTask implements Runnable {
    protected final Session                     notesSession;
    protected final Collection<String>          callBackMessages;
    private SessionCloner                   sessionCloner;
    private NSFComponentModule              module;
    public AbstractNotesBackgroundTask(final Session optionalSession, final Collection<String> messageHandler) {
        this.callBackMessages = messageHandler;
        // optionalSession MUST be NULL when this should run in a thread, contain a session when
        // the class is running in the same thread as it was constructed
        this.notesSession = optionalSession;

    public void run() {
        this.callBackMessages.add("Background task run starting: " + this.getClass().toString());
        try {
            Session session;
            if (this.notesSession == null) {
                NotesContext context = new NotesContext(this.module);
                session = this.sessionCloner.getSession();
            } else {
                // We run in an established session
                session = this.notesSession;
            /* Do the work here */
        } catch (Throwable e) {
        } finally {
            if (this.notesSession == null) {
                try {
                } catch (NotesException e1) {
        this.callBackMessages.add("Background task run completed: " + this.getClass().toString());

    private void setDominoContextCloner() {
        // Domino stuff to be able to get a cloned session
        if (this.notesSession == null) {
            try {
                this.module = NotesContext.getCurrent().getModule();
                this.sessionCloner = SessionCloner.getSessionCloner();
            } catch (Exception e) {
    protected abstract void runNotes();
You only need to subclass it and implement runNotes() to get started. However you probably want to hand over more parameters, so you need to modify the constructor accordingly. Please note: you can't bring Notes objects across thread boundaries. Next the class that controls all the threads. It should live in the session or application.


XPiNC development insights

QuickImage Category
I am developing a rather large XPiNC application for IBM internal use. One application and parameter database pushes data around 10 backend databases. Part of the databases might be local (depending on the user), some might be on the server and not all users have access to all databases. Most of my business logic lives in Java code, both in classes, beans and managed beans. In the course of the development I stumbled over a series of insights a XPiNC developer should be aware of. Here it goes, in no specific order:
  • FacesContext.getCurrentInstance() might return null when called in your Java class, depending on when you call it. I had calls to the FacesContext in some of my managed bean property calls which failed on page load with a Error 500 that the standard error page couldn't trap. You need the FacesContext to resolve any variable e.g. getting a handle on a different managed bean. In conclusion I rearchitected my beans to have only one managed bean per context modeled as Facade pattern that provides a single access point to all classes needed
  • The ExtlibUtil.getCurrentSession() uses the FacesContext under the hood, so there is a distinct risk of getting a null session. So using the dependency injection pattern all calls requiring a NotesSession have a NotesSession as calling parameter. This requires a bean design that is populated elsewhere (e.g. DataContext, ObjectDataSource), so you can have getters and setters without parameters
  • Longer running operations (stuff we typically do in an agent) will time out. You don't have the luxury in XPiNC to use agent.runOnServer(), so you need a different approach (and anyway I despise mixing XPages and agents). With the help of a friend and the ever impeccable Dublin team I figured a way to run Notes backend accessing code as a thread (subject to another story)
  • Debugging is way less comfortable than in XPages. You have to put the whole client into debug mode, not just connect to the http task. So I tend to start my Notes client with -RPARAM -consoleso I see what System.out and System.err are producing. I also tend to keep my code in Java to make it more testable easily
  • The XPages Debug Toolbar works well in XPiNC
  • There is no way to restart the XPiNC task like you can restart the http task or the web preview task in Designer. When you use external jars and update them, you have to restart your client to test it successfully
  • In a data source you can specify databases by replica id. This is nice since you might not know where a local replica is located. However you only can specify the replica id and unless NotesDatabase.openByReplicaId() not the server. So you never know which database you might get (local or one of the known server). So I compute once (and keep it in the sessionScope) the actual filepath, trying local first, and use server/filepath instead of replica id for data sources
  • The Java Collection framework is your friend. You can use them nicely as source for a repeat control. Just keep in mind: a list will give you the entry in your instance variable, while a map will give you they key only. Of course a key can be any Java object, so some creative key creation goes a long way
  • A very popular way in Notes applications was the use of NotesDocument.computeWithForm(). When connecting to existing applications you will be tempted (or actually required) to use it too. When the computeWithForm tries to trigger some UI interaction in the @Formula, your XPiNC tab will lose focus to a native tab (I haven't figured out which one gets selected) and a externalContext.redirect() will simply not happen
  • In an XPiNC application sessionScope and applicationScope are local to the current user. So status exchange via applicationScope won't work in XPiNC. After all you run a local JEE server instance in your local Expeditor framework (a.k.a Notes client). This even applies when you load a server based NSF (hardly a good idea in XPiNC). This changes when you configure XPiNC to run on the server, then the applicationScope is for all users.
  • Since XPiNC applications load all Java classes from the NSF into the JEE server memory, its best to load them from a local NSF, even for applications that need to have current data for all users. Here XPages' ability to use data from a different database comes in handy and you would load your data NSF from the server - I'll prepare some code snippets for future posts
As usual YMMV


Managed Beans, XPages and Testability

QuickImage Category
I like my Java to be well managed, properly prepared and of top quality.
Unless you are living under an XRock, you know that managed beans are (one of) the talk of the town in the XPages community.
While managed beans are simple beans that have been given a name and a scope, they need some thought when you want to test your applications.
Refering from one bean to another or to a session and the current database is made easy using However I like to test my code outside the XPages environment, where such a call (or any call to a JSF resolver) will fail. So I came up with a helper class to make the beans testable. Let's start with a typical managed bean setup:
<?xml version="1.0" encoding="UTF-8"?>
<!--AUTOGEN-START-BUILDER: Automatically generated by IBM Domino Designer. Do not modify.-->
<!--AUTOGEN-END-BUILDER: End of automatically generated section-->
To access those beans from plain Java as well as the session and current database I use a helper class. It doesn't matter if you use thee helper from the command line or in an XPages, the results will be the same. What is important: you can't store the oobjects when you use them in the XPage since they are either not thread save (session, database) or would loose their scope association (the rest). If you want to make the setBeans (see below) method more robust, you could call javax.faces.context.FacesContext.getCurrentInstance(). If anything else but null is coming back, you are in the XPages thread and should not set the objects to anything but null (or add that to your getter?)
The enum.INSTANCE is a Singleton and asures that I always get the same object back. This allows me to write a test class (or actually use JUnit tests) like this:

import lotus.domino.Database;
import lotus.domino.NotesException;
import lotus.domino.NotesFactory;
import lotus.domino.NotesThread;
import lotus.domino.Session;

public class LocalTest {
public static void main(String args) throws NotesException {
System.out.println("Starting local test");
Session s = NotesFactory.createSession();
Database db = s.getDatabase("", "Customers\\swg\\ctpsi-ap.nsf");
String roles = "[Scout]", "[Naseweis]", "[Neffe]";
Huey huey = (Huey) DuckFactory.makeDuck("Huey");
Dewey dewey = (Dewey) DuckFactory.makeDuck("Dewey");
Louie louie = (Louie) DuckFactory.makeDuck("Louie");
DuckFetcher.INSTANCE.setBeans(s, db, huey, dewey, louie, roles);
// Ducks do what ducks have to do
DuckAdeventure d = new DuckAdventure();
// Game over
DuckFetcher.INSTANCE.setBeans(null, null, null, null, null);
The test class is easily run from Eclipse or the debug perspecitve in Domino Designer and you can step through with the debugger. Other than the local http preview the code runs with the current userid's permissions, so you don't need to mess with the ACL. The whole class is about a hundred lines and pretty easy to understand:


Don't try this at home!

QuickImage Category
Domino has a refined security system, so the java.policy file can be a real PITA. So you would be tempted to write a few lines of LotusScript and run it on a scheduled agent, so on the next server restart that pain goes away. Of course you wouldn't write code like below which lacks any error handling.You also would not hide this code from your admin people who would want an impact study and your firstborn for any change they make. So instead of doing all this you wait until there is a proper configuration setting for this.
Option Public
Option Declare

Sub Initialize
    Dim s As New NotesSession
    Dim inikey As String
    Dim secFileName As String
    Dim stream As NotesStream
    Dim policy As String
    Dim beginString As String
    Dim endString As String
    Dim permission As String
    Dim beginPos As Integer
    Dim endPos As Integer
    inikey = "NotesProgram"
    beginString = "grant {"
    endString = "}"
    secFileName = s.Getenvironmentstring(iniKey, true)
    secFileName = secFileName + "jvm/lib/security/java.policy"
    permission = "permission;"
    Set stream = s.Createstream()
    Call stream.Open(secFileName)
    policy = stream.Readtext()
    beginPos = InStr(policy,beginString)
    If beginPos < 1 Then
        'We don't have any so we abort
        Exit sub
    End If
    Dim firstCut As String
    firstCut = Mid$(policy,beginPos)
    endPos = InStr(firstCut,endString)
    If endPos < 1 Then
        'The file is borked
        Exit Sub
    End If
    Dim allGrant As String
    allGrant = Mid$(firstCut,1,endPos)
    'Now the check
    If InStr(allGrant,permission) < 1 Then
        'We need to update the file
        Call stream.Truncate()
        Call stream.Writetext(Mid$(policy,1,beginPos+7), EOL_NONE)
        Call stream.Writetext(permission, EOL_PLATFORM)
        Call stream.Writetext(Mid$(policy,beginPos+7), EOL_NONE)
    End If
    Call stream.Close()
End Sub
As usual YMMV


Modernizing Notes applications - lessons from the trenches

QuickImage Category
Not only since mobile first became fashionable corporations are trying to ditch the Lotus IBM Notes client - for various reasons.
These efforts were branded "modernization", "web enablement", "mobile enablement" or if a competitor had a word "migration". Initially there was hope that this would be a short, painless and automated process (the upgrades, not the migrations that is). But reality taught a few facts that you need to consider:
  • A Rich Client is based on RichText, a browser client on HTML. There is no 1:1 mapping (otherwise one format would be superfluous), only some approximation (just ask Ben about it). Much more: there is no 1:1 mapping of the event models and APIs. So any automation would lead to incredible hacks
  • The Notes client's and Domino server's LotusScript runtimes are, while dated, incredibly robust and forgiving. The amount of OMG code I've seen in Notes applications (including my own ) tops any other platform.
    Still that code gets the job done and out of the way. However that doesn't translate into another language in an automated fashion, but rather demands the repayment of quite some technical debt
  • Code that gets a client with one user huffing and puffing can't simply be expected to run for hundreds or thousands of users on a single server
  • Users expect web (and mobile) applications to be fresh and modern. If Jack Sparrow couldn't find the fountain of youth, how can an automated tool do that?
    This is a clear conflict of interest between the users ("shall be modern and fresh") and the ones paying for it ("make it work in a browser, fast & cheap") - a classic for Why enterprise software sucks
    John D. Head clearly outlines expectations and possibilities for modern user interfaces. Teamstudio provides a nice set of modern mobile controls (that even work offline for a fee)
  • The amount of Notes applications mostly get underestimated greatly. Use a scientific method to create evidence
So damn if you don't, damn if you do? Not quite. In an approach Peter calls Asymmetric Modernization you step back from the tree to see the forest. Instead of looking the usual Notes way "application-by-application", see the sum of the applications and wipe them out, all of them modernize them all in one go (it is called economy of scale).
Nathan and Peter share a video, the modernization of the nifty-fifty, a recent case study and the service offering. Go check them out.
The biggest issue I see with this approach is the usual cautious stand in IT today:"let us do one (insignificant) application first and see how it goes. Then we linearly extrapolate and get scared" That is the total opposite of Asymmetric Modernization, so it will require clever persuasion to get a project approved.
As usual YMMV


Planning applications (XPages MindMap)

QuickImage Category
In a recent XPages workshop in Kuala Lumpur, the class brainstormed on the planning process for an XPages application. This is what we came up with. For every item on the list one could elaborate quite a bit, but putting that on the map would make it rather messy.
Planning XPages MindMap


We love Java and JavaScript

QuickImage Category
But there are the other too...

Comment from Oliver:"That's not originally mine. I never made a cartoon about LotusScript. I guess it originally was about VBA (or the like). Someone had forked the original cartoon. Why not ;) "


It ain't pretty, but this is what we do!

QuickImage Category
Recycle your Domino objects, or a kitten must die!


The IBM Collaboration Olympics have started

QuickImage Category
There are 2 contest starting around IBM Connections and IBM XPages:
The XPages contest
The Connections contest
If you like to develop software, join them. Even if you feel you are not up to compete, at least cheer them. The contest have yieled excellent results we all benefit from!


Mobile Application Interaction Models

Latest since Eric Schmidt announced the mobile first doctrine in Barcelona, every developer knows that it is coming.
Of course with the fragmentation of the runtimes (think Android, i/OS, Blackberry, Bada, Windows Phone 8 etc.) and the development platforms (Objective C, C++, Java, C#) the discussion rages on: is a web application (think HTML, CSS, JavaScript) sufficient or do I really need to write native code for each platform? I covered my view on the options before.
At a closer look, the difference is not so much about how an application is developed, but about the interaction model used. Of course each development environment leans towards a specific interaction model. Web applications tend to interact online, while native applications can do anything, but work tentatively offline (think Angry Birds)
One obvious problem with online applications is network coverage (think: everywhere, just not in this conference room or plane), another perceived one is bandwidth. The situation is improving and obscures the real issue: Latency. An online application works roughly like this:
Mobile Application with network traffic
The data packages are, once the main page is loaded, actually quite small, but frequent.
How to explain latency? I usually take the restaurant as example. Bandwidth is the size of the tray the waiter can carry. In a small bistro the waiter needs to go back and forth a number of times if you and your football team (soccer for our American friends) order their beers at once - you have a bandwidth problem - one that doesn't exist on the Oktoberfest. Latency however is the time from calling the waitress until she appears and the time she needs to place the order. Now imagine instead of ordering your 11 beers in one go, you do that sequentially one by one (besides p*****g the waitress off), you have a latency problem. The waitress spends more time running back and forth than actually serving beer.
Network Latency is the issue here
As long as you sit in a well connected environment, you won't experience much:
  • My server in the home network has a latency of about 0.2 ms
  • In the IBM office the local servers have about 6-8ms (Switches & Firewalls take their toll)
  • when I reach out to an overseas server that is not cached by a CDN I get latencies of 200-300ms
  • On mobile 2 or 3G in a crowded place that latency easily goes up to 1.5sec
Now imagine you have an application that makes 100 small Ajax calls like populating a dropbox, doing a typeahead, load contact details etc.
  • In my local network that amounts to a total delay of 20ms, not noticeable
  • In the office it is still below a second
  • overseas it is already 30sec
  • on the patchy mobile network 2.5 minutes, rendering such an application useless.
The solution here, available to native and web applications is to stick to rule 1 of application development:

Keep the network out of the user experience!

This sounds easier and harder than it actually is:
Mobile Application with local interaction
The UI would only interact with local content and all network communication would, as far as possible, happen in the background. HTML5 knows the storage API and is quite offline capable. Background operations are supported using web workers or (via Titanium) MQTT (my personal favourite).
Of course that mode is much harder to master - which applies to the native applications too: suddenly you have data on a central server and a mobile device and need to keep them in sync.
This is manageable for One-user-access applications like email, but quite a headache for concurrent access applications like Collaboration, ERP or CRM. Short of reinventing Notes replication (someone else did that) you could treat the local data as "cache only" and queue updates. Of course proper sync needs extra meta data, quite a headache for an RDBMS.


Explaining web enablement challenges to business users

QuickImage Category  
With XPages Notes and Domino application can be the new sexy and run beautifully on all sorts of devices big and small. So a whole cottage industry (no insult intended) of offerings around Domino Application Modernization appeared.
Modernization always also means: browser and mobile enablement.
Expectations ran high, that a magic button would transform (pun intended) a code base organically grown over two decades into beautiful working responsive web 2.0 applications. But GIGO stands firm and not all applications are created equal. Domino's (and LotusScript's) greatest strength, turned into a curse: being an incredible forgiving environment. Any clobbered together code would somehow still run and lots of applications truly deserve the label "contains Frankencode".
There is a lot of technical debt than needs to be paid.
The biggest obstacle I've come across is the wild mix of the front-end (a.k.a Notes client) and back-end (core database operations) in forms views and libraries. This problem never arises in the popular web environments, since there are different languages at the front and back at work (e.g. JavaScript/PHP, JavaScript/Ruby, JavaScript/Java) - only in very modern environments it is all JavaScript (the single language idea Notes sported 20 years ago).
The first thing I taught every developer in LotusScript, is to keep front- and backend separate and keep the business logic in script libraries that only contain back-end classes. Developers who followed these guidelines have a comparable easy time to web enable application.
But how to explain this problem to a business user (who probably saw some advertisement about automatic conversion to web, be it on IBM technology or a competitor)?
Tell them a story (if they are not interested at listening at any of that, there's a solution too)!
Here we go:
You are supply specialist for natural resources exploration company and your current assignment is to get your geo engineers set up in a remote jungle location. So you have to source vehicles, build roads and establish a supply chain. Probably you get a bunch of those (a living legent since 1948), stock spare parts and ensure that you have diesel stations along the way.
Everything is fine - the road might be a little patchy here and there, but that's not a problem, you get you guys delivered and working. You even look good (sometimes).
This are your Notes client applications, delivering business value, robust, efficient and can deal with a lot of road deficiency (that would be code quality).
Your remote location becomes successful and suddenly the requirements change. People want to get there in style (Browsers). Your gas stations will do, no problem here, but already the roads need to be a little less patchy and your stock of spare parts and the mechanics trained on them are useless. That would be your front-end classes and the "mix-them-all-up" coding style that worked in the past.
If the "arrive-in-style" meme escalates further (mobile devices) you need to build flawless roads (unless your oil has been found in Dallas where proper roads supposedly exist).
An experienced supply planner might anticipate what is coming and while sending in the Unimogs already prepare the gravel foundation, so paving of the road for the fragile cars is just a small step. Or nothing has been done for a while and the healthroad check comes back with a huge bill of material.
You get the gist, now go and tell your own story.


Mastering XPages released - in Chinese

Chinese companies are one of the fastest growing adopters of XPages. Being cut of from a lot of blogs by the Great Firewall of China, having a good reference for XPages is essential for their success. So a team from the China development lab (CDL) teamed up to translate the standard reference into Chinese and make it available to all the smart engineers behind the GFW.
XPages Team
Congratulations to the team!

XPages in Chinese


Testing your mobile applications on real devices

QuickImage Category
Unless you are hiding under a rock, you develop mobile XPages applications. Since emulators only get you so far (the iOS emulator only runs on Mac where you don't have a Domino Designer), you want to test the app on the real devices (after you convinced your boss to buy all them). For "always on" developers with access to a development server that isn't an issue. When you are "on the run" and all you have is Domino Designer, it just doesn't work. When you try to connect to your Domino Designer "Preview in Web Browser" from anything else than localhost ( or ::1) you will get an Error 500 Access violation.
With a little creative configuration you can get around it! Review this post and you (almost) know what to do. Instead of "proxy mode" you use "listener" mode, let's say port 88. Configure the target address to Port 80. Once set you can connect to your HTTP preview from "outside" on port 88. This can be the iPhone emulator on your OS/X host talking to the Domino Designer Web in your Windows VM or any mobile phone talking to your local real or virtual machine. A little step for a developer... Of course: local preview doesn't magically support authentication (I tried really hard hacking the local names.nsf), so you need a real server to test that one.
As usual YMMV


Extracting data from Domino into PDF using XSLT and XSL:FO (Part 4)

QuickImage Category  
This entry is part of the series Domino, XSL:FO and XSLT that dives into the use of XSLT and XSL:FO in IBM Lotus Domino.

So far we had a look at the all over process, some Java to convert FO into PDF and FO Basics. Time to get some Notes data exported. To make the task easier I created a little helper that allows convenient export into XML. Currently it works on a NotesDocument(Collection) basis, but it would be a small step to add a method that uses a view navigator for speed.
The most complete rendering of a NotesDocument is done using the .renderXML method. For a collection you use a NotesDxlExporter. Unfortunatelty both methods are comparable slow. So I added an alternate approach (works only if you don't have RichText for the moment) and export lean XML.
A Form2XMLDefinition class (optional) allows to pick which fields need to be picked in the XML file. It also allows to group those fields (more on that another time - or look at the source). So the methods are:
package com.notessensei.fop;

import lotus.domino.DocumentCollection;
import lotus.domino.Session;

public interface Notes2XML {
    public abstract void addForm(Form2XMLDefinition newForm);
    public abstract ByteArrayOutputStream renderDocument2DXL(lotus.domino.Document doc);
    public abstract ByteArrayOutputStream renderDocument2XML(lotus.domino.Document doc);
    public abstract ByteArrayOutputStream renderDocumentCollection2DXL(Session s, DocumentCollection dc);
    public abstract ByteArrayOutputStream renderDocumentCollection2XML(DocumentCollection dc, String rootName);
I expanded the PDFReport class with a convenience method: getNotesXMLExporter() to be able to reuse my managed bean (beginnings of a Facade pattern).


Extracting data from Domino into PDF using XSLT and XSL:FO (Part 3)

QuickImage Category  
This entry is part of the series Domino, XSL:FO and XSLT that dives into the use of XSLT and XSL:FO in IBM Lotus Domino.

In Part 2 I introduced the Java code required to output PDF from XSL:FO and how that code can be called from an XAgent. Now lets have a look at XSL:FO itself. It is a W3C defined standard to layout documents. You could consider it as competitor to PostScript or PDF. XSL:FO contains both layout instructions and content. Since it is expressed entirely in XML, it is easy to manipulate (follow me for a moment and accept XML is easy) and - more importantly easy to split content and layout for reuse. Typically a complete XSL:FO document would be only an intermediate step in PDF production. The report design (without data) would be contained in an XSLT stylesheet that gets merged with XML data. You could consider XSLT the "templating language" of XSL:FO.
A XSL:FO document has a single <fo:root> element. This contains one or more page-sequence elements, that contain the actual content and a layout-master-set, that defines the pages.
XSL:FO layout-master-set
Besides the page size (and content orientation) a simple-page-master defines header, footer, left and right column (called regions). You need to get your math right there. The margin and the regions are both substracted from the page size to compute the real margins. When you have a margin="1cm" and a region-start with 3cm width, then the left margin is 4cm. Read up the XSL:FO tutorial on for more details.
The main element region-body allows to specify a column-count attribute that will create multi-column page layouts without the need for a table and a manual calculation of column content. You also could define alternating page masters, like left and right pages or different pages for a chapter beginning - read details in the repeatable-page-master-alternative specification.
Your main content is contained in one or more page-sequences.
The page sequence contains the content. Don't get confused: a page-sequence represents content of n number of pages (n >=1), not just one page. You need more than one page sequence only when you want the page layout to use a different master/style (of course using the alternatives mechanism described above you can achieve alternate styles inside a single page sequence). The page sequence contains one or more flows. A flow is targeted at a region (there are 5 of them) and contains block elements (think HTML div,p,table etc.) that contain the content. There are a huge number of specialised attributes and elements (stuff like watermarks or graphics) available you can learn about in the specifications.
In practise you write a sample XSL:FO document and transform it into PDF. Once you are satisfied with the results, you then convert the XSL:FO document into an XSLT stylesheet. This is easier than it sounds, you simply wrap xsl:template tags around your fo and replace your sample content with xsl:apply-templates statements. w3schools has a simple example. Of course XSLT is a interesting topic on its own, go and read the XSLT 2.0 and XPath 2.0 Programmer's Reference (also available on Kindle).
Next stop: How to pull Notes data into XML for processing.


Extracting data from Domino into PDF using XSLT and XSL:FO (Part 2)

QuickImage Category  
This entry is part of the series Domino, XSL:FO and XSLT that dives into the use of XSLT and XSL:FO in IBM Lotus Domino.

In Part 1 I discussed the process of getting data from Domino to PDF using only open standards. In this part I want to start backwards: If I have valid XSL:FO, how do I get the PDF output? In a later installment I will discuss what XSL:FO is capable of and how to create it using XSLT. Finally I will discuss how to extract XML from your data source (Domino or others).
I choose the backwards approach since it is easier to understand walking from the desired output towards a stylesheet than creating a stylesheet if the result isn't clear. Bear with me for this approach.
I will use Apache FOP as my rendering engine. The Quickstart compresses the needed steps neatly into 1-2-3 (pun intended):
  1. Download
    This is the easy part. Pick one of the mirror servers and get (24M) - check if you read this later if there is a newer version. Extract it to a directory of your choice, we will import/copy what we need from there
  2. Configure
    You have choices:
    • copy the Fop files into jvm/lib/ext (bad idea)
    • import it into an NSF (twice: once for agents, once for XPages)
    • create a plug-in (good for sidebar and XPages, not good for agents)
    Having coined the term XAgent I will stick to the XPages version with import into the NSF. Time permitting I'll add a plug-in approach to this little series.
  3. Run
    The FOP website provides a good overview on general Java use as well as servlet specifics. Using that information as template it is not too hard to implement a managed bean that takes a XMLDocument and an optional Stylesheet and returns the rendered PDF as result
Since our output will be rendered by a managed bean, we need to configure it in the faces-config.xml:
The XAgent follows the usual pattern:
var exCon = facesContext.getExternalContext();
var response = exCon.getResponse();
var out = response.getOutputStream();
response.setHeader("Content-disposition","inline; filename=result.pdf");
response.setHeader("Cache-Control", "no-cache");
// In this example the text in sessionScope.longText will be rendered
var written = sessionScope.longText != "" ? "<text>"+sessionScope.longText+"</text>" : null;
// Writes the default rendering out
// Stop the page from further processing;
To get the Java class working I needed to import:
  1. avalon-framework-4.2.0.jar
  2. batik-all-1.7.jar
  3. commons-io-1.3.1.jar
  4. fop.jar
  5. serializer-2.7.0.jar
  6. xmlgraphics-commons-1.4.jar
The class for the PDFReport contains just some wrapper code to render "FO" or "XML with XSLT" into a PDF. It is demo code, you want to add more robust error handling. Next stop: more about FO
As usual YMMV.


Extracting data from Domino into PDF using XSLT and XSL:FO (Part 1)

QuickImage Category  
This entry is part of the series Domino, XSL:FO and XSLT that dives into the use of XSLT and XSL:FO in IBM Lotus Domino.

We all know "Notes doesn't print". Nevertheless the topic of document output and reports is not going away, even if I'd like to ban the reports. There are plenty of ready made tools, but today I'd like to start with home cooked reporting.
Why the effort? Using only tools that use open standards you gain more control over the whole process and you can use whatever deems fit. The downside: it is more things you need to know and might not be suitable for business users (but its great to torture interns). In the long run you have a portfolio of source transformations that you can combine potentially faster than any reporting tool. The general principle is "Extract Transform Render":
Extract Transform Render
  1. Extract:
    Whatever will pull out the XML for the second step will do the trick. For list type of rendering ?ReadViewEntries will do the trick or simple DXL exports. Quite often you might opt for some bespoke code to extract code with an eye of a fast and/or easy transformation phase. You also might consider to extract your data in conformance with an established international standard
  2. Transform:
    This step usually takes the XML from the extract phase and runs it through one or more XSLT transformations. XSLT is kind of IT Black Magic (other say it's just set theory) and can use quite some computing power. For high performance the pros use a dedicated applicance. Once you get the heck of XPath you can do some amazing reporting (e.g. "give me all sales guys where withing the last 5 sales of the 3 guys next to his ranking there was a carpenter")
  3. Render:
    Rendering is easy. The outcome of the transformation step will be XSL:FO which is a page description language. Use a free renderer or a commercial offering and a few lines of code. The output typically is a PDF file, but you can target graphic formats too.


Setting focus to the first invalid field

QuickImage Category
When it comes to validation, the only secure way is server side validation (Client side validation is more: "nice suggestions for users"). One little grievance developers have - and try to fix with elaborate measures - is the requirement to set the focus on the first failed field. XPages makes this quite simple. As Tommy pointed out fields that are invalidated server side get the attribute aria-invalid="true".
Besides, as Tommy suggested, using it to style the input (check his comment - cool idea), you can use it to set the focus on the first invalid field. The key component is the dojo.query function. It allows all sorts of interesting queries, including for attributes. So add a simple Script block with this static value to your page:
XSP.addOnLoad(function() {
     var invalid = dojo.query("[aria-invalid=true]");
     if (invalid.length > 0) {
As usual YMMV


Reuse web agents that use PRINT - Bean edition

QuickImage Category
Using Java in XPages does have performance advantages especially when you can use EL instead of SSJS. So I created the Java version of the reuse web agents post. You need to copy the Java either into your NSF or include it in your own extension library. Calling an agent is limited to the same database as the XPage resides in, I leave it to an exercise for my readers to improve on that.
You define the Java class as a managed bean:
and then simply include (e.g. in the value of a computed field) ${agentHelper.agentResult("SampleAgent")} as EL entry. You have to be careful to ensure your agent doesn't run twice. The agent side adjustments (the LotusScript stuff) is the same as for the SSJS version. The Java version is slightly more flexible since it allows String or Stream results and acting on a new or existing document.


Webservices in XPages - AXIS vs. CXF

QuickImage Category
I wrote about using Apache AXIS to connect to a web service before. It worked like a charm when you import the AXIS library into the NSF. In the past few days I tried to move this code into an extension library and I had initially very little success. Here is what I tried and what finally worked:
  1. The AXIS libraries are provided as a plug-in in the Domino server, so my first take was to declare a dependency in my plug-in to that plug-in. To successfully create a Extension Library I had to declare dependencies to and to be able to extend Unfortunately the plug-ins expose Java commons logging, so the classloader complaint and didn't load the AXIS classes
  2. Second attempt was to split the code into two plug-ins: one that depended on the AXIS plug-in and another one that depended on the former and and This didn't yield any better result
  3. 3rd test was to import the AXIS jars into a plug-in and only depend on and but not the AXIS plug-in. Didn't work either
  4. Finally I tried to switch from AXIS to the newer and more flexible Apache CXF. CXF is able to provide transport over many protocols, not only SOAP and REST but JMS and others too. Initially I felt it was too complex for my little requirement, especially since the list of jar dependencies is quite long.
    Turns out CXF is the optimal solution. I used CXF 2.5.2 and the provided wsdl2java utility. The command line I used is [path-to-cxf-install]/bin/wsdl2java -frontend jaxws21 -client your.wsdl. The -client parameter is optional, but generates a nice sample client that shows the call parameters for all web service methods.
    I found I either need to include the WSDL file in the jar or point to the URL where the WSDL file is available online to get CXF to run. Now for the best part: All dependencies including CXF are available in Domino (I tested on 8.5.3), so other than the Java code generated I didn't had to import, depend etc. on anything (it actually might work even in a Java agent). So CXF is the way to go.
Update: As Bernd pointed out in the comments, it is important to update the permissions ( [Notes program directroy]/jvm/lib/security/java.policy ):
grant {
permission java.lang.RuntimePermission "setContextClassLoader";
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";

For testing I used a dictionary service that returns the definitions found for a given word. My class (the part I had to write myself) is rather short:
package demo;

import java.util.List;
import javax.xml.namespace.QName;

public class Dict2Test {
    public final static String WORD_TO_LOOK_FOR = "Trust";
    public final static String WSDL_URL = "";
    public static final QName SERVICE_NAME = new QName("", "DictService");

    public static void main(String[] args) throws IOException {
        Dict2Test dt = new Dict2Test();
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        dt.getWordInfo(out, WORD_TO_LOOK_FOR);

    public void getWordInfo(OutputStream out, String theWord) {
        OutputStreamWriter w;
        DictService ds;
        DictServiceSoap dss;
        w = new OutputStreamWriter(out);
        try {
            ds = new DictService(new URL(WSDL_URL),SERVICE_NAME);
            dss = ds.getDictServiceSoap();
            WordDefinition wd = dss.define(theWord);
            List<Definition> allDef = wd.getDefinitions().getDefinition();
            if (allDef.isEmpty()) {
                w.append("<h1>No definition found for ");
            } else {
                w.append("<h1>You were looking for: ");
                for (Definition oneD : allDef) {
        } catch (Exception e) {
            try {
                w.append("<h1>" + e.getMessage() + "</h1>");
            } catch (IOException e1) {
            } finally {
                try {
                } catch (IOException ex) {
To test the class I defined it as a managed bean in faces-config.xml:
and created an XAgent with the following code


Make Java code for XAgents easy to test

QuickImage Category
One of the popular entries in this blog is my introduction into XAgents together with related considerations and practial applications. There are a few patterns, that make XAgents easier to develop and debug.
Despite outstanding tool contribution debugging in XPages is still wanting, so any code that you can incorporate pre-debugged is a bonus. Here is what I do:
  • I try to write my logic in Java. I use the SSJS only to fetch the values I need to call my Java function
  • The functions of the Java class take always the OutputStream (or the ResponseWriter) as a parameter. This is where the output goes
  • For rendering XML or HTML output I use SAX with a little helper (Making one of these for JSON would be an interesting exercise)
  • I try to avoid using XPages specific (e.g. variable resolver) in my class containing the logic, but rather rely on Dependency Injection to ease testing
  • I debug and test the Java parts outside of XPages
A little sample. Imagine I have a class that renders a Notes document into a PDF (with a fixed style). In my XPages code (beforeRenderResponse) I would write:
// The external context and the response object
var exCon = facesContext.getExternalContext();
var response = exCon.getResponse();

// Deliver uncached PDF
response.setHeader("Cache-Control", "no-cache");
// Force file name and save dialog
response.setHeader("Content-Disposition", "attachment; filename=invoice.pdf");

// The Output Stream for Binary Output
var out =response.getOutputStream();
var pdfProcessor = new;

// Here the call where "document" is a Notes document
pdfProcessor.renderInvoice(out, document);

// Done
Not really rocket science. The beauty of this approach: I can have a standalone Java application to test my PDFFormatter class. I use Eclipse for that. In Eclipse I have configured the Notes client's JVM as the runtime for the project. This way all Notes Java classes are found. Most likely you need the Notes program directory on your system path (but you know that). A simple testing class can look like this:
public class Tester {

    public static String theURL = "Notes:///8525744100531C7D/F4B82FBB75E942A6852566AC0037F284/034DAEE1CEEE2FB58525744000719185";

    public static void main(String[] args) throws Exception {
        Session s = NotesFactory.createSession();
        Document document = (Document)session.resolve(theURL);
        PDFFormatter pdfProcessor = new;
        FileOutputStream out = new FileOutputStream(new File("invoice.pdf"));
        pdfProcessor.renderInvoice(out, document);
Of course that approach works too when you plan to use JUnit or a similar framework. Finally all can be glued together with ANT for a full automated environment, but that's a story for another time.
As usual: YMMV


Reuse web agents that PRINT to the browser in XPages

QuickImage Category
When upgrading classic Domino applications to XPages one particular problem arises constantly: "what to do with the PRINT statements in existing agents that write back directly to the browser?" Currently there is no automatic way to capture this output.
However with a little refactoring of the agent the output can be recycled. You can use a computed field for the result showing it on a page that maintains the all over layout of your new application or use the XAgent approach to replace the whole screen (I'm not discussing the merits of that here). These are the steps:
  1. Make sure your agent is set to "Run as web user" in the agent properties
  2. Add the AgentSupport LotusScript library to your agent
  3. Initialize the ResultHandler class: Dim result as ResultHandler
    SET result = new ResultHandler
    that will take in the print statements
  4. Use Search & Replace in your agent and replace Print  with result.prn 
  5. Add at the end call
  6. In your XPage or CustomControl add the AgentSupportX SSJS library
  7. Get the result with a call to agentResult("[name-of-your-agent]"). You can process it further or display in in a computed field etc.
The sample code is for illustration only. You want to add proper error handling to it. My test XPage looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="">
        <xp:script src="/AgentSupportX.jss" clientSide="false"></xp:script>
    This comes from an agent:
    <xp:text escape="false" id="computedField1">
The sample agent like this:
Option Public
Option Declare

Use "AgentSupport"

Dim result As ResultHandler

Sub Initialize
    Set result = New ResultHandler
    result.prt "Some message"
    result.prt "<h1> a header </h1>"
End Sub
Of course the interesting part are the two script libraries.


Preparing for a boring flight - offline (Extract media from a feed)

QuickImage Category
David Leedy provides us with the incredible useful Notes in 9 (a.k.a tutorials and insights about XPages and Notes. The feed with all the videos is hosted by feedburner. To enjoy them while off the grid you can subscribe to them using iTunes, but that's for Warmduscher .
I'll show you how to use curl and a shell script (easy to translate to a cmd file):
  1. First you download the feed: curl -G -L -o notesin9.xml
  2. Run the transformation: xslt notesin9.xml feedburner2curl.xslt (on Windows you would use .cmd)
  3. Make the script executable: chmod +x
  4. Fetch the movies: ./
This technique works for any media RSS feed (ATOM wouldn need a different XSLT), so it is worth to be added to the toolbox. There are a few moving parts (which you should have anyway): You need to have curl and a XSLT shell script (that uses a jar file) as well as the stylesheet to convert the feed into a command file. The XSLT command file looks like this:
notify-send -t 500 -u low -i gtk-dialog-info "Transforming $1 with $2 into $3 ..."
java -cp /home/stw/bin/saxon9he.jar net.sf.saxon.Transform -t -s:$1 -xsl:$2 -o:$3
notify-send -t 1000 -u low -i gtk-dialog-info "Extraction into $3 done!"
(where only the line with "java..." is relevant, the rest is eye candy). The XSLT stylesheet isn't much more complicated (the statements are in one line each, so check the download version to get them right):
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl=""
    <xsl:output indent="no" method="text"/>
    <xsl:template match="/">#!/bin/bash<xsl:apply-templates select="//media:content" /></xsl:template>
    <xsl:template match="media:content">
        curl -C - -G <xsl:value-of select="@url"/> -L -o <xsl:value-of select="reverse(tokenize(@url,'/'))[1]"/>
The only interesting part is reverse(tokenize(@url,'/'))[1] which I use to get the file name - basically the String after the last /. "tokenize" and "reverse" need a XSLT 2.0 processor.
Update: Got a little carried away and used the wrong script, it only did XSLT 1.0, now corrected using Saxon and XSLT 2.0. Thx Ulrich for noticing.
As usual YMMV


Give your web applications a speedboost and HTML5 compatibility boost in Internet Explorer

QuickImage Category
With the accelerating emergence of mobile devices HTML5 takes centre stage. While the first round of the "browser wars 2.0" was all about speed, the second round is about HTML5 compatibility and behaviour. With the dominance of Android and iOS in the mobile market Microsoft's Internet Explorer isn't the gold standard for browsers anymore, the test results rather show IE as trying to catch up. Furthermore, since it isn't a standard if it doesn't allow for divergent interpretation, Microsoft implements HTML5 features different than the rest, creating a headache for developers. The irony in the whole situation is, that Microsoft won "browser wars 1.0" because the implemented HTML more faithful that its then competitors.
Quite some organisations claim (with various levels of credibility) to be stuck/standardised on IE (which I usually reply to with: "Interesting, what version of IE is running on your CEO's iPad?"), so you can't use all the new fancy stuff you could use on the mobile devices (until Windows Phone7 becomes popular and the "IE does it differently" headache starts on mobile). Luckily there is a cure: Chrome Frame. It installs inside IE and doesn't load any pages until you tell it so. So all the legacy code can still run with IE, but your new stuff uses Chrome's webkit rendering engine, solves a lot of XPages/Dojo headaches too. There are a few steps to get going:
  1. Add <meta http-equiv="X-UA-Compatible" content="chrome=1"> to your HTML head section. This will ensure Chrome Frame is activated when rendering your page. If all of your applications on your (Domino) server should use it, you also can use a HTTP header (that's the web/Internet Sites section in the Domino directory) X-UA-Compatible: chrome=1
  2. Add discovery and download options to your page(s)
  3. If you are the system administrator, you want to download the MSI Installer and push it out (I like OCS Inventory for that). Make sure you read the admin guide
You can install it now. As usual YMMV.


Learning XPages

QuickImage Category
This entry is my reference for the collection of learning materials around XPages and adjacent technologies. I get asked a lot and in the spirit of sharing I make my recommendations public. There are a number of technologies and skills touching XPages, so it looks like a long list, but you don't need them all to get started.


When deleting XPages watch your form property "Display XPage instead"

QuickImage Category
During the XPages training in Manila we found an interesting behaviour of the "Display XPage instead" property of a form. To reproduce this follow these steps:
  1. Create a form form1
  2. Create an XPage test1.xsp
  3. Set the form property "Display XPage instead" (both for Client and web) to test1.xsp
  4. Preview form1 in Notes client and web - works as designed, test1.xsp will show.
  5. Now delete test1.xsp
  6. Preview form1 in Notes client and web - it will show an error
  7. Inspect the "Display XPage instead" property of form1: it will show empty since test1.xsp is no longer a selectable item, However that value is still stored in the form (we didn't touch it at this point)
  8. Open the form1 as DXL. At the end of the form you will find:
    <item name='$XPageAlt'><text>test1.xsp</text></item>
    <item name='$XPageAltClient'><text>test1.xsp</text></item>
  9. To fix this: Either delete the lines in DXL (if you dare ) - or: in the form property "Display XPage instead" select any other XPage (optional) and then the empty item on top and save the form. Just open and save the form doesn't do the trick, you have to touch the form property
Nice one


Balsamiq Mockups in XPages

QuickImage Category
I write most of my technical blogs to document solutions of problems I have encountered in "real" applications. The popular XAgents post is no exception here. Then I was investigating how to make Confluence plug-ins work in XPages. The specific plug-in I had in mind was Balsamiq Mockups, the tool that has proven invaluable for any screen design, both here on the blog as well as in countless customer discussions. Once I figured the XAgent it was rather easy:

If you want to embed Balsamiq Mockups in your own XPages application, reach out to Peldi or Valerie and buy the Balsamiq Plug-in for Confluence.
Update: To make it very clear: Using the Confluence plug-in in XPages is not covered by the Balsamiq license for Confluence, so you need to individually negotiate with Peldi and clearly indicate that you want to run that in XPages. You also, for the time being, need to promise not go go back for support for any challenge that can't be verified in a Confluence installation.
Update 2: Peldi and I are talking an there will be a solution. Stay tuned


Upcoming XPages Training 2011

QuickImage Category
There are 68 days left in 2011 and quite some of them will be filled with XPages trainings and chat. Here is the current list of my public XPages sessions in case you want to join:
  • 7/8 November 2011, XPages class, Melbourne Australia (with Lotus beer 8th evening). Register with Geoff Cates (event full, still feel free to join for the LotusBeer)
  • 9 Nov 2011: available in Melbourne for 1:1 discussions (coordinate with Geoff Cates)
  • 10 Nov 2011: availabe in Canberra for 1:1 discussions (coordinate with Stuart Hickson)
  • 11 Nov 2011: available in Sydney for 1:1 discussions (coordinate with Stuart)
  • 24/25 November 2011, XPages class, Singapore. Register online
  • 01/02 15/16 December 2011, XPages class, Manila Philippines. Register online
  • n.n. Dec 2011, Shanghai, China. Register with Xu Gang
Given demand we probably can squeeze in one or two more locations, so pester ask nicely your local IBMer to organize a session.
See you there!


XML Helper class for XAgents

QuickImage Category
When rendering your own output in XPages agent style (a.k.a XAgents) you might want to render XML as output. While you easily can try String concatenations to get your output, you will very fast hit the wall once you deal with special characters and double byte. One way to render proper XML is building a DOM Document. This can get very memory intensive if you have a large output. The alternative is to use the SAX Parser as I recommended before. The only catch is that you have to take care of XML nesting yourself. Since I use that quite often I created a little helper class long ago, that makes XML output a snap. Here is a sample XAgent code:
var exCon = facesContext.getExternalContext();
var response = exCon.getResponse();
var out:javax.servlet.ServletOutputStream = response.getOutputStream();
var d:biz.taoconsulting.xmltools.SimpleXMLDoc d = new biz.taoconsulting.xmltools.SimpleXMLDoc();
// Top level
// Inside Demo
d.addSimpleTag("Client", "ACME Inc");
d.addSimpleTag("Location", "RoadRunner Valley");
// And done
resulting in this XML send to the caller:
<?xml version="1.0" encoding="UTF-8"?>
  <Client>ACME Inc</Client>
  <Location>RoadRunner Valley</Location>
The helper class has methods for: Opening Tags, simple Tags, empty Tags, CDATA content and closing a specific number of tags. It supports DocTypes and a stylesheet instruction but currently no namespaces. Have fun with it.
As usual: YMMV


Documenting Validation in XPages

QuickImage Category
The last thing in a developer's mind is documentation since developers believe that hardly anyone reads it as long as nothing goes wrong or needs to be changed. What works comparable well is the generation of documentation out of code. JavaDoc, JSDoc and LSDoc are examples for such tools. They work well when the code has little external dependencies. When validating form submissions however documenting what field gets validated would be missing. I'm advocating to use validators rather than onSubmit code, since they allow to generate this documentation easily (at least when you wrap your head around XML and XSLT). Since an XPage or a custom control is XML we can use XSLT to generate documentation (how exactly the transformation could run is subject to a future post). You can end with a report like this:
Validator Documentation
The (not covering all types of validators yet) stylesheet looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl=""
    <xd:doc scope="stylesheet">
            <xd:p><xd:b>Created on:</xd:b> Oct 17, 2011</xd:p>
            <xd:p><xd:b>Author:</xd:b> Stephan H. Wissel</xd:p>
    <xsl:output method="html" indent="yes"/>
    <xsl:param name="filename">XPage</xsl:param>
    <xsl:template match="/">
                <title>Validators for <xsl:value-of select="$filename"/></title>
                <link rel="stylesheet" href="style.css" type="text/css" media="screen" />
                <h1>Validators for <xsl:value-of select="$filename"/></h1>
                        <xsl:apply-templates select="descendant::*[xp:this.validators]"></xsl:apply-templates>

<xsl:template match="xp:this.validators">
    <xsl:variable name="rowspan" select="count(child::*)+1"></xsl:variable>
    <tr class="fieldname">
        <td  rowspan="{$rowspan}" class="fieldname"><xsl:value-of select="ancestor::*[1]/@id"/></td>
        <td class="fieldtype" colspan="3">
             <xsl:value-of select="name(ancestor::*[1])"/> -
            <xsl:value-of select="count(child::*)"/> validator(s)
    <xsl:apply-templates />
    <xsl:template match="xp:validateRequired">
        <tr class="validator">
            <td class="validatortype">Required</td>
            <td class="validatorcode"></td>
            <td class="validatormessage"><xsl:value-of select="@message"/></td>
    <xsl:template match="xp:validateExpression">
            <tr class="validator">
                <td class="validatortype">Expression</td>
                <td class="validatorcode"><xsl:value-of select="xp:this.expression"/></td>
                <td class="validatormessage"><xsl:value-of select="@message"/></td>
    <xsl:template match="xp:validateLength">
        <tr class="validator">
            <td class="validatortype">Length</td>
            <td class="validatorcode">min: <xsl:value-of select="@minimum"/> max: <xsl:value-of select="@maximum"/></td>
            <td class="validatormessage"><xsl:value-of select="@message"/></td>

    <!-- Catch validators we don't know exclude types as you add them above-->
    <xsl:template match="xp:this.validators/*[name(.) != 'xp:validateLength' and name(.) != 'xp:validateRequired' and name(.) != 'xp:validateExpression']">
        <tr class="validator">
            <td class="validatortype"><xsl:value-of select="name(.)"/></td>
            <td class="validatorcode">please check source code!</td>
            <td class="validatormessage"><xsl:value-of select="@message"/></td>
As usual YMMV


One or two click mobile approvals?

QuickImage Category
With Mobile First being the parole of the day more emphasis needs to be spend creating good mobile user interfaces. As much as the web is not print, mobile is not just a small desktop. Nevertheless we apply lessons learned from desktop interfaces to mobile devices and learn as we go along. Especially corporate development managers dream of write once, run everywhere (while we know that the best achievable case is write once, test everywhere). Some of the critical differences:
  • Your screen is small
  • Your pointer is big
  • Your network is unpredictable
  • You want to swish
  • You aim is weak (especially when in a bus, train, plane or ship)
  • You rather select than type
  • You really want to focus on essentials only
So your UI needs to cater for that. On the other hand you could just have one UI based on mobile that you use on the desktop too. However short of using a widget tile, that approach looks silly and is less productive than a well designed desktop interface. One interesting facet surfaced this morning when discussing a mobile approval application:
Approval with 2 buttons Approval with a radio and one button
Is the typical pattern with an Approve and a Reject button the right approach for mobile? Surly it would feature one click decisions, a god sent for time plagued managers, of course only until you hit the wrong button due to a road bump. So alternate approaches are needed. One as pictured above would be to have a radio button (or its mobile equivalent) and the button. Another possibility would be to switch from the request screen to one that lets the decision maker add a comment and finalise the answer:
Finalizing the answer
Still the challenge remains what contextual information to show with a request, but that's a story for another time.


Designing data sources - square pegs into round holes

QuickImage Category
Stephen Mitchel translated the end of verse 21 of the Dao De Ching ( 何以知衆甫之狀哉?以此。= wu he yi zhi zhong fu zhi zhuang zai? yi ci.) as "How do I know this is true? I look inside myself and see.". Lao zhu could have been a software platform or framework architect . Getting the gist of a platform often requires a zen like approach. Unfortunately in the heat of delivery pressures things get lost and we end up with "Frankenworks" instead of "Frameworks", where functionality works, but feels rather "bolted on".
Currently there is work underway to make XPages a first class RDBMS front-end. The data source looks very promising, nevertheless prompted me to reflect on the nature of data access, like I mused about structures before. There are some structural differences between a document centric approach (Domino, XMLDB, XForms, ObjectDB, JsonDB etc) and a relational database. Doing justice to both sides poses a formidable challenge:
  • The nature of a relational database is a flat set. A set of rows and columns that get created, read, updated and deleted. It is always about a set, that more than often can yield from more than one table. There is no such thing as a single record, it is just a set with one member. All OR-Mappers struggle with this
  • The nature of Domino the the document. Data is stored in documents. Collections (views/folders/search/all) are designed to get access to a document (set). The result of this nature is a dual access to data: there is the collection which is read only (and can be flat or hierarchical) and there is the document which is read/write where data changes happen.
  • The document has a predefined set of meta data absent from a relational table: ID, access control, various dates, hierarchy (isResponse) etc. One could add those to an individual database schema, but they can't be taken for granted in RDBMS (a story for another time: designing a RDBMS schema to work well with XPages)
  • The document sports structured data. In Domino that are multi-value fields, in other NoSQL databases these structures can be more complex. In RDBMS these structures are splattered across multiple tables and pulled back together with JOIN statements. This makes it easy to run reports or do mass-updates, but makes transporting a logical entity from one database to another a pain
  • The dominating clause in RDBMS is WHERE which is needed for all operations including updates, while Domino acts on the current document (
  • The document is closely connected to the Notes/Domino event model. Both XPages and the classic Notes client (and to a lesser extend classic Domino) offer rich data events:queryNewDocument, queryOpenDocument, postOpenDocument, querySaveDocument, postSaveDocument etc.
  • SQL doesn't provide an event model, but the various RDBMS implementation provide triggers that serve a similar purpose that run stored procedures (and are mostly written in incompatible flavors of SQL - check SwissQL for translating them). I'm sure about INSERT, UPDATE and DELETE triggers as equivalent to query/post save events. I haven't looked for a while, but last time I checked SELECT wouldn't trigger a stored procedure, but you could call one directly.
  • The splattering of data across tables in RDBMS led naturally to another capability of relational databases, that comes in handy for large manipulations too: transactional integrity. If all you need is saving one document, there is no imminent need for a transaction mechanism, distributing data over multiple (parent, child) tables however mandates an integrity protection
Interestingly a document oriented data model is closer to the real world's (read business users') perception of data: a contract is a document, as is a purchase order as is a bank note. Tables usually serve as (table of sic.) content listing for documents or detail listings inside documents.
So what does that mean for the design of additional data sources in Domino? There are two possible approaches which luckily can co-exist since they are not mutually exclusive: follow the nature of Domino or follow the nature of the source (not the nature of the force, that's for others). The current OpenNTF extlib approach is the later: it is designed around the relational feature set.
Going forward I would like to see data sources that build on the duality of the Domino data access: the read-only collection and the read/write document.
Datasources and the API
  • Each data source will have 2 elements: a read-only collection and a read/write entry/record/document
  • These sources have the same method signatures as DominoDocument and DominoView. So in a design a developer could swap them out for each other. MyRDBMSSource.getItemValueString("Location") would work the same way as DominoDocument.getItemValueString("Location"). For an RDBMS developer that might look a little strange, but only caries a one time learning affordance, greatly outweighted by the benefit of swappable sources. Of course the parameters would be rather different. In a RDBMS source there probably would be a parameter to define what getDocumentUniqueID would return.
  • All the document events would fire with every data source
  • Data sources can implement additional tags to offer access matching their nature
What data sources can I imagine? This is not a revelation of IBM's plans, I rather expect some of them being provided by the community or as commercial business partner offering:
  • enhanced JDBC source following the Domino pattern
  • Domino data source encapsulating the inversion of logging pattern
  • DB/2 PureXML data source. It would use the standard JDBC approach for the collections and PureXML to read/write document data. It would implement the spirit of NSFDB2 without the constraints of replicating all NSF features (data only)
  • Sharepoint. One could build Sharepoint front-ends that survive a Sharepoint upgrade without the need to rewrite them
  • IBM MQ
  • Web services (take a WSDL and make a form)
  • CouchDB
  • 3270 Terminal / IBM HATS
  • HTML5 storage
What's your imagination? (need help?)


Scaling XPages across servers

QuickImage Category
In China everything is bigger. The wall is longer then elsewhere and companies, catering to 1.3 billion people have a lot of employees. I'm currently working with our experts from the China development lab to figure out, based on discussions with customers and business partners, how to scale XPages when one server is not enough.
When it comes to architecture, opinions are a dime a dozend. We had "experts" chipping in who would happily split XPages server and NSF server into two introducing network latency between the two. Others are convinced, that only RDBMS is the real thing. Since Domino 8.5.3 happily connects to an RDBMS, the question became more interesting (not that we didn't have options before). As usual the devil is in the details, which in case of Domino would be Multi-Value fields and Reader and Author protection.
We are planning to have 2 servers and test them in various configurations to see whar are the performance ramifications:
  1. XPages and NSF Server

    XPages and NSF Server
  2. XPages Cluster

    XPages Cluster
  3. XPages and RDBMS

    XPages and RDBMS
It might take a while to get results, and I'm very curious which opinion withstands the bright light of evidence. Of course just 2 servers won't deliver evidence for a whole farm, but it is a start.


Custom Controls in XPages

QuickImage Category
Well designed Custom Controls (we just got 64 new ones) and a set of good libraries can accelerate XPages development tremendously. Of course preparation takes time and it takes some enlightenment to be ready to invest the time to get them right. As a quote from the London Developer Co-op states: "Dirty still remains when Quick is long gone". Some of my insights into custom controls (from various stages of interacting with them):
  • Custom Controls are like subforms
  • Custom Controls have parameters
  • Custom Controls allow a higher level of abstraction
  • There are better ways than currentDocument for Data
  • There can only be ONE: Custom Controls and HTML ID
  • Parameters cut both ways
  • Knowing Dojo helps
  • JavaScript is interpreted - changable at runtime
  • Writing code that writes code is like Inception
  • The JSF and HTML life cyle is important
  • Sharing custom controls is easy
  • Custom Controls are fun!
What makes a custom control
This is the summary of my AusLUG presentation.
Update: The session is concluded, I posted the slide show on prezi and below enjoy:


More on spreadsheets in XPages

Dennis really was busy. Given the fact, that he only touched XPages a month ago, the result is impressive
Click on the image to see the control in action
Soon to be found in the GPL catalog of OpenNTF. Commercial support will be available from Potix.


Java security in XPages

QuickImage Category
Once you push the boundaries of XPages (which are very elastic) you sooner or later will run into Java security. The Java security settings live in the file
[notes program]/jvm/lib/security/java.policy. The text file (don't edit it in Symphony, use gEdit or Notepad) defines a series of grant codebase "somelocation" { Java Permission } entries that govern what code located at a specific location is allowed to do and what not. The general rule: it must be allowed, otherwise it is forbidden. The basis entry grant { ... } defines what all Java classes can do. You definitely want to be very stingy here. Lazy developers (like me) add to this file:
grant { permission; };
... which is of course a big No No for a production environment (but a big Yes Yes for a productive dev environment - waiting for the howls). When you are ready to deploy your Java infested enhanced nsf, you need to update the java.policy file on your target machine (waiting for another howl) - unless you were smart and packaged your Java into an extension library which has the right level of security and is easy to deploy). The syntax is easy (but not necessarily logical for us mere mortals):
grant codeBase "xspnsf://server:0/yourdatabase.nsf/-" {

Jim clarifies: you only need to change yourdatabase.nsf, the other parts are static, directory separators are / regardless of platform, the location is relative to the data directory and the whole codeBase value must be lower case (regardless of your file names). Of course you can limit the permission to what you actually need. Finding out what that is when you use ready baked libraries can be a little tedious and warrants its own future post.


Coming soon to an XPage near you

Dennis Chen has been busy preparing for his submission to the XPages contest. It looks very promising:
Coming to an XPage near you soon
and lean:
<zk:zssapp width="100%" height="100%" src="ZSS-demo_sample.xlsx"></zk:zssapp>


What XPages developer are you?

QuickImage Category
There are 11 types of XPages developers, but you must understand binary to get that joke. There is an ongoing discussion what does it take to be an XPages developer and what skills are required. Regardless what development platform you use, web development is much harder than client development. You are dealing with runtimes you can't control (a.k.a what browser users use), network uncertainties (will that application be accessed via a shaky GSM connection or in a high latency network?) and a confusing set of languages that play together but have their own query mechanisms and syntax quirks: HTML, CSS, JavaScript. In a client only environment you usually get away with one language. Most modern IDE try to shield developers from this complexity but at some point in time are all subject to a leaky abstraction, so it is better to learn the languages of your trade. Nevertheless depending on one's personal aptitude there are different roles to be filled when developing XPages:
There are 3 types of XPages developers
  • Develop XPages - a.k.a Phil and gang.
    The team that makes XPages possible. Deep rooted in Java, JSF, NAPI and JNI they provide us with the SSJS engine, the XPages runtime and the implementation of various standards like JSF or JavaScript (a.k.a ECMA Script / ISO/IEC 16262). Besides occasionally writing a book they provide the core set of generic custom controls
  • Develop for XPages - a.k.a The Nathan & Niklas club.
    They extend the platform, build mobile extensions, provide an application transformer or run custom control contests. They understand XPages deeply, including its roots in JSF and the control life cycle. They code both in Java and JavaScript and have some dojo for breakfast
  • Develop with XPages - a.k.a. The rest of us.
    We understand users' needs, business requirements, workflow processes and the corporate development life cycle. We know how to distinguish between users' needs and wants and how to translate a vague descriptions of "make it work" into a concrete application design. Our favourite mode of operation is to drag & drop ready made custom controls and sprinkle Pixi dust SSJS between them as glue. We don't have time for the deep understanding both other groups need. We love JavaScript since we don't need to mentally reset when switching between front-end and back-end code (something our JSP, PHP and ASP peers envy us for). We are the largest group, but possibly the most quiet one too. For quite some of us developing software is just a way of making a living
So where are you and where do you want to be?


Deploying the extension library - client edition

QuickImage Category
Unless you are living under an XRock your next XPages project will use the Extension Library. While installing it into your development environment is easy (just follow the instructions) and deployment to your 8.5.2 servers got a lot easier, deploying it to XPiNC application takes a few steps more:
  1. You need an update site. Since you deployed it to the servers you would have one already
  2. You need a Widget catalog (typically called toolbox.nsf). Luckily that's just a File - Application - New
  3. You need an entry in the widget catalog for the extension library plug-in with a widget.xml describing the plug-in (details below)
  4. You need a client policy that points all Notes clients to the Widget catalog and makes the category you assigned to the Extlib Widget an automatic mandatory install
  5. Sit back and relax and see your extension library deployed
You can muse if you update site should use HTTP or NRPC. The later would have the advantage, that you can replicate the update site to your clients (using a policy) making the update process independent from actual network connections. The tricky part is to get the XML file right.


4 Validation errors to avoid and some fun with XPages Validation

QuickImage Category
Input validation is a tricky business (on any platform). On one hand you need to ensure that your form contains sufficient and accurate data to be processes as intended. On the other hand validation needs to get out of the user's attempts to get things done. XPages offers client and serverside validation. While you could continue to use onSubmit or QuerySave the recommended approach is the use of validators. They nicely separate the validation logic from the validation trigger and from the error display. This gives you a great deal of flexibility. You can go to the extend to provide your own Java validators that can be used by you application developers without knowing Java, add your own message programmatically or let your code raise an error. Per summarised error management nicely and Andre explains better SSJS validation rules. So we are pretty much covered. Nevertheless I see quite some worst-practises in action:
  1. Only client side validation: Client side validation is for the comfort of your user. It is to make her life easier. Client side validation is not suitable for preserving data integrity! Using Firebug, NoScript, cURL or simply notepad/textedit/gedit (to create a simple html form with method="post") a user can bypass any client side validation. So it must be accompanied with server side validation code (unless of course all your users are angels)
  2. Abrasive error messages: Sounds familiar: "Input error: This is not a phone number", "You must fill xxx", "Wrong character detected"? Users translate (conscious or subconscious) those messages into "You stupid moron, you not even can fill in the form - serve me, I'm your digital overlord". Guess what happens next: an eMail, a phone call and application avoidance. The better way is to a) be more liberal with what you accept (phone number can contain spaces, plus, minus and slash - and how hard is it to filter out spaces from a credit card number? b) Let the computer apologise for its short comings: "Sorry you need to complete all indicated fields before this application can process your request" or (if you buy into the idea that users attribute personality to computers and applications): "Sorry I only understand credit card numbers that have no spaces"
  3. Lack of save as draft: If your form has very few fields only, you can get away without. But any modest complex application will get a user to a point where he is stuck with a missing piece of information (or a weekend (s)he wants to start). So you need to accept input however incomplete it might be, just don't process it yet. Since an eMail system on the market has a "draft mode" your users will be both familiar with it and actually expect a save as draft function
  4. Validation traps (I'm not talking about a Catch 22): fields are validated on field exit and on failure the user is prompted and send back to that field. While it might be well meaning (make sure everything is fine) is is actually evil. It disrupts the user in her workflow and forces a certain working style on them. For a better way to use onBlur see below
To summarise:
Validation always must be server side, it should be augmented with client side validation for the comfort of the user and in XPages should be implemented using Validators.
(Should in the context of this statement means: "Do this unless you have 7 good reasons to do it differently".)
Does that mean that you have to code every validation twice? The short answer: No if you use Validators. I imagined a modification of the XPages engine, where you could trigger a validation onBlur of a field that would only hint the user about the problem (so NO messagebox prompt) and not require to duplicate code that you use in your validator. In a chat with Máire Kehoe she enlightened me how this can be done in XPages, so I present the ValidationHelperControl


A custom control that should be on top of every page

QuickImage Category
This idea is shamelessly ripped from an XPages chat with Nathan who will contribute a <crashie6 ../> control to the Extension library soon. Go to the navigator view and create these 2 files in custom controls :
The xsp:
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="">
  <!--[if lt IE 7]>
 <div style='border: 1px solid #F7941D; background: #FEEFDA; text-align: center; clear: both; height: 75px; position: relative;'>
   <div style='position: absolute; right: 3px; top: 3px; font-family: courier new; font-weight: bold;'><a href='#' onclick='"none"; return false;'><img src='' style='border: none;' alt='Close this notice'/></a></div>
   <div style='width: 640px; margin: 0 auto; text-align: left; padding: 0; overflow: hidden; color: black;'>
     <div style='width: 75px; float: left;'><img src='' alt='Warning!'/></div>
     <div style='width: 275px; float: left; font-family: Arial, sans-serif;'>
       <div style='font-size: 14px; font-weight: bold; margin-top: 12px;'>You are using an outdated browser</div>
       <div style='font-size: 12px; margin-top: 6px; line-height: 12px;'>For a better experience using this site, please upgrade to a modern web browser.</div>
     <div style='width: 75px; float: left;'><a href='' target='_blank'><img src='' style='border: none;' alt='Get Firefox 3.5'/></a></div>
     <div style='width: 75px; float: left;'><a href='' target='_blank'><img src='' style='border: none;' alt='Get Internet Explorer 8'/></a></div>
     <div style='width: 73px; float: left;'><a href='' target='_blank'><img src='' style='border: none;' alt='Get Safari 4'/></a></div>
     <div style='float: left;'><a href='' target='_blank'><img src='' style='border: none;' alt='Get Google Chrome'/></a></div>


Add a "Collapse All" button/link to your categorized view

QuickImage Category
Quick tip for today. The XPages View control allows to display categorized views. While I don't like them in web applications one still wants to use them. In the Notes client categorized views have a handy "expand all" / "collapse all" SmartIcon pair which is missing in XPages. The shortest way to get these is to add an output script to your page:
XSP.attachPartial("#{id:viewColumn1}__shrink:1", "#{id:collapseAllElement}", "#{id:viewPanel1}", "onclick", function(){}, 0, "#{id:viewPanel1}_OUTER_TABLE");
You need to adjust the names in bold red. Works like a charm. Left to the reader: figure out the command for Expand all (or create a toggle control).
As usual YMMV.


What's your Data Definition Language?

QuickImage Category   
After recent insights in data structures I was wondering what's the right format to describe data models. Its a thought almost alien to Notes developers since we "just add what we need". Nevertheless having a data model eases maintenance and documentation (which brings up the question what comes first: the application or the data model). Clarity about data models also fosters the contract first way of developing applications where agreements about interfaces and data structures are made up-front before implementation. There are a number of contestants available to choose from (with no claim to completeness):
  • SQL Data Definition Language. Basically that's all the CREATE TABLE statements you use to create your RDBMS tables and views. Advantage of SQL DDL is its closeness to RDBMS, which makes implementing the described data easy, the ability to create the definition in a simple text editor but also rich visual tools (like ERwin which was one of the first of its kind) and the capablity of other DDL (like UML) to read/write SQL DDL. The biggest drawback in a world where SQL no longer rules alone is its closeness to RDBMS and its lack of support for transmittable data (think web service, sync or two folded MVC pattern). Today I would say SQL DDL is hardly the source of your data model anymore, but an output from one of the other DDLs (most likely UML)
  • UML and its XML representation. The Unified Modeling Language is designed to do much more than describing data. Besides data one can model other structures (like components, deployments or packages), behaviours and interactions. There is a huge offering of UML tools available on the market: Rational System Architect, Visual Paradigm, DIA (or Visio ), Violet, UMlet, Altova UModel (Windows only) and many more. UML can output almost any other format including export to XML Schema. There is also a lot of literature available about this topic.
    While UML certainly covers all aspects of modelling it doesn't come without drawbacks. Working with UML does require a dedicated tool or plug-in, so it is hard to quickly getting stuff done (graphics can get in the way of execution speed). UML offers 2 ways to model data: Entity Relationship Diagrams and Object Diagrams. None of them really fit the document working style we find in web services or web 2.0 applications, so some of the XML Schema generated look a bit "hammered into shape". Furthermore I haven't seen a lot of tooling that would verify data to conformance with an UML model at runtime.
  • Eclipse EMF models. EMF has been designed mainly as a modelling framework to generate code (and thus other data definitions) out of its models. EMF data itself is stored as XML and there is an impressive list of documentation available online and in print. Nathan is a big EMF fan. One would most likely work in Eclipse to work with EMF, I haven't explored the suitability for text editing. There also are a lot of transformations available from and to EMF, which are pending my attention.
  • XML Schema (including RELAX.NG and DTD). When you are comfortable reading and writing XML, using XML Schema comes natural. While you perfectly well can write it in a plain text editor, you most likely will use at least an XML aware editor like oXygen XML, Stylus Studio, XMLSpy (which all run inside Eclipse and are Schema aware), Eclipse's base XML Editor or any other of the countless offerings. If you deal with SOA you will realise, that WSDL, the contract language of SOA, uses XML Schema in its bowels. XML Schema also can be used to validate documents on the fly without the need to first generate code out of it. I like the capability to define my own data types and to mix and match existing Schemas to fit my specific needs. Custom data types would be classes/objects in UML and are (to my knowledge) absent in SQL DDL. Since XML Schemas live at a (usually public) URL and there are a lot of existing Schemas available already (diverse topics like music or eBusiness (UBL) etc) duplication of efforts can be reduced. XML Schema can be transformed to SQL (I would of course rather suggest to use PureXML, but that's a story for another day)
  • I didn't find any tool to model/verify JSON data. Since JSON is by definition schema free. However I expect, that some generic JSON schema will appear over time, the declaration driven validation is too valuable.
Looking at Notes and Domino, the XML nature of XPages, ATOM and RDF, Activity Streams and the IBM Social Business Toolkit I made XML Schema my first choice, however I'm ready to be convinced otherwise.
What is your Data Definition Language?


Caching Lookup results in XPages

QuickImage Category
Nathan and John have shown before how a column value in an XPages view can contain the result from a lookup into another view (Notes or RDBMS). As you all know "With great powers ....". These lookups are executed once per row in the view. If there is a possibility, that the lookup key is repeating, it is a good idea to cache these values. Depending on your application this can be the one of the scopes: requestScope, viewScope, sessionScope or applicationScope. A very simple cache mechanism can look like this:
var myLookups = {  
    "getColor" : function(color, name) {
        var fruit;
        if (requestScope.fruit != null) {
            fruit = requestScope.fruit;
        } else {
                fruit = new java.util.HashMap();
        if (fruit.containsKey(color)) {
            var result = fruit.get(color);
            return result;
        var lookResult = @DbLookup(@DbName(),"Fruits", color,3);
        requestScope.fruit = fruit;
        return @Trim(@Replace(lookResult,name,""));
The function can be called with myLookups.getColor("red","Apple"); (the function returns all fruits of a given color except the current fruit handed over as the second parameter). If you have a lot of variables to cache a HashMap might not get the best result and you could consider using JCS. To test if it is working alter the function like this (Line 12+13):


Burncharts in XPages

QuickImage Category
One little step for a developer, a big step for agile software development: Julian has implemented my dojo Burncharts into the latest release of the xCharting project on OpenNTF. Have a look at the demo and use it happily. Nice work Julian.
Update: Gave Julian back his real name (Ave Julius is quite outdated isn't it)


Binding controls to managed beans

QuickImage Category
Out of the box XPages connect to Notes Views and Notes Documents as data sources. Other data souces require some advanced skills you might not be ready to invest. However controls can be bound to many things, data sources are just one of them. Under the hood the data binding uses the JSF Expression Language (EL). Since XPages knows a JavaScript binding you can bind your control to any JavaScript function, but that's not what I want to show today. One very interesting option (that is 1:1 JSF) is to bind your controls to a managed bean. So what is a managed bean? It sounds more complicated than it is. The bean part is a standard Java class that has a constructor without parameter (which is the default for a Java class anyway) and get/set methods pairs (more about that in just a moment). The managed part is an entry in the faces-config.xml where you define a "friendly" name for a Java class as well as the scope. If you use that friendly name in code (in SSJS or EL) the class instance is automatically created for the defined scope if it doesn't exist yet in that scope. Once the scope expires the object is automatically discarded. Since this works automagically the Java class is called "managed". An entry in your faces-config.xml (to find that file, open the Navigator view and look for /WebContent/WEB-INF/ in your NSF) could look like this:
With this entry a new top level object demo is available for use in EL or SSJS in the same way as you can use database, session, context etc. Our class is valid for the session context, so multiple XPages will see the same content. You can name your methods inside the Java class as you like and call them in SSJS, e.g. demo.playTune(). The more interesting use however is the bean use. If your (real live) object has e.g. 3 properties: Name, Color, Taste, then you would have 6 methods in Java, 2 each for each property: getName(), setName(newName), getColor(), setColor(newColor), getTaste(), setTaste(newTaste).
Once you follow this pattern you can refer to that property in EL just by its, demo.color, demo.taste (Note: no brackets here!). You enter that in XPages data binding in "Advanced / Expression Language". It will however appear in the regular data binding setting with the bean's friendly name as name of a data source.
A little stumbling block: the properties start always lowercase, while the first letter after get/set must be uppercase. Also the get function needs to return the same data type as the set function takes in as a parameter.
XPages inspects the bean for the get/set functions. If the set function is missing a control is automatically rendered read only, while a missing set function leaves an entry field empty. Once data is posted back to the server the set functions of your bean are called and it is up to you what to do with the data. Having named get/set function is great when you exactly know your object at design time. A very typical use for a bean binding however is retrieving a data set where you don't know what is coming. EL also can cater for that. Your Java class needs to implement a property that returns a Java MAP in a pair. e.g. Map<String, String> getStuffProperties(); void setStuffProperties(Map<String, String> newProps); Once you have that you can write as data binding in EL:demo.stuffProperties["Color"]. Here you have the freedom to use upper- or lowercase property names as you like. Of course you are not limited to a static string and have any expression EL is capable of at your disposal. This makes a quite powerful construct. Try it yourself! Use this simple Java class:
package com.demo;

import java.util.HashMap;
import java.util.Map;

public class ELTest {

    private String color;
    private String taste;
    private String name;
    private Map<String, String> fruitProperties = new HashMap<String,String>();

    public String getColor() {
        return color;

    public void setColor(String color) {
        this.color = color;
        this.fruitProperties.put("Color", color);

    public String getTaste() {
        return taste;

    public void setTaste(String taste) {
        this.taste = taste;
        this.fruitProperties.put("Taste", taste);

    public String getName() {
        return name;

    public void setName(String name) { = name;
        this.fruitProperties.put("Name", name);

    public Map<String, String> getFruitProperties() {
        return this.fruitProperties;

    public void setFruitProperties(Map<String, String> fruitProperties) {
        this.fruitProperties = fruitProperties;


Learning XPages' foundations: JSF and Dojo (including the Expression Language)

When people ask me what exactly XPages is/are, I answer with the formula:
XPages = JSF + JavaScript + Notes + Domino Designer - J2EE headaches
The J2EE headaches would be XML descriptor files, deployments and the lack of an integrated database. Nevertheless a large part of the power of XPages results from its foundation in JSF. So understanding JSF surly doesn't harm. O'Reilly provides in their Safari Books Online platform a online video training with more than 10 hours duration: Building Ajaxified Web Applications with JSF 2.0 by Marty Hall. In 15 lessons you learn a lot about JSF and Ajax.
  1. Overview
  2. Getting Started
  3. Basics
  4. ManagedBeans
  5. Navigation
  6. Expression Language
  7. Properties
  8. Handling GUI Events
  9. AJAX
  10. Validation
  11. Looping
  12. Data Tables
  13. Templating
  14. Composite Components
  15. View Params
Probably a lot of the functions a JSF developer needs to take care of are handled by Domino Designer in XPages, but it is good to know what the platform handles for you to appreciate even more. Knowing the inner workings will enable one to take full advantage of all capabilities.


Rendering your own output in XPages revisited

QuickImage Category
My blog post Web Agents XPages style proves to be particularity popular with a lot of questions coming in. There are a few clarifications that I'd like to share:
  • You can have one: either the Writer (for text) or the Stream (for binary content - sample here). You actually might want to opt for the stream even for textual content depending how you create it (see below)
  • You call the faceContext.getResponseWriter() in the afterRenderResponse event, while you call the facesContext.getOutputStream() in the beforeRenderResponse event (remember only one please!)
  • It seems you don't need the writer.close(); at the end. Starting with 8.5.2 this actually throws an error
  • A smart way to render output is actually a Java bean that takes a stream as a parameter. Such a bean can be fully tested and debugged outside the XPages context
  • Very often the use of your own content is to output XML or HTML or a combination of both. In LotusScript you simply use Print, so the temptation is rather high to use writer.write("..."). This misses an opportunity to take advantage of code other people have written and debugged. Let a SAX Transformer take care of encoding, name spaces and attribute handling. Use this sample as a starting point (yes SAX can be used to create a document - use a Stack to control nesting). If you have a complicated result you could use a DOM object to render your result which has the advantage that you can assemble it in a non-linear fashion
  • You can easily add an XSLT transformation to the output. This increases flexibility but costs performance, so it is a good idea to keep the Stylesheets precompiled in an application bean.
  • When you need to output XML from 2 different name spaces (e.g. XML and HTML) you can either use SAX's name space feature or you add the HTML in text sections. SAX takes care of encoding and add CDATA staements
As usual YMMV (and let me know how it goes). Other posts on rendering your own output in XPages:


What is holding back XPages adoption

QuickImage Category
A question that is asked - monstly by non-developers or "old Notes hands" - quite often recently: "Why would I want to develop in XPages rather than classic Notes and Domino". Part of that question might be driven by Cainotophobia. However it is valid. When a new way of programming comes around, like switching from punch-cards to terminals, from Assembler to C, from Cobol to Basic, from VB to or from Terminal to GUI draught of competency happens. The existing paradigm is well understood (including all its quirks) and backed by a huge body of knowledge represented by a huge code base. The transition from classic Domino to XPages is no different.
The transition between LotusScript and XPages
Until the new paradigm is backed by a similar level of competencies and skills doubt will exist if "that new thing" is really the way to go. With increasing adoption of XPages the skill level in classic Domino will gently decline (people retire or simply forget) while XPages skills will rise. Now the graph for XPages isn't fully accurate once you take developers into account who haven't developed on Domino before. After all XPages embraces and (horrible IBM word:) leverages existing technologies and open standards. Unless you live under a rock you have (at least as a consumer of the results) encountered HTML, CSS and JavaScript by now. Also XPages' foundation in JSF is well established. Of couese it very much depends on the "readiness for innovation" in the individual companies' development team if they want to contribute to the growing body of knowledge around XPages or wait until the moment where late adopters move. IMHO XPages' early adopter phase is coming to an end and XPages enters Domino mainstream. Very soon the question will change from "Why XPages" to "Why stay back".


Application Configuration / Parameters in XPages

QuickImage Category
Any application of reasonable size has parameters that configure it. Be it the salutations, the names of departments or the task priorities. In classic Notes and Domino application you typically find @DBColumn, @DBLookup or @GetProfileField to retrieve these values. Short of @GetProfileField you could do the same in an XPages application. However I would advocate a different approach. These are the steps:
  1. Create a SSJS script library. Call it "ApplicationConfiguration[ForNameOfYourApp[Group ] ] "
  2. Create one JavaScript object (see source below), call it "[ NameOfApp ]Config" (e.g companyConfig)
  3. Create inside this object functions that start with get and the Parameter you want to retrieve (e.g. getDepartments)
  4. In your application you then include that library and simply use the object with its function (e.g. companyConfig.getDepartments() )
What you do inside that object isn't that important for this approach. It is of course important for maintainability, performance etc., but that's a story for another time. So you can stick to your @DBColumn or @DBLookup for starters (until you learn better). So why would you want to follow that approach? There are 2 reasons: abstraction and flexibility. Inside your XPage you have an uniform way to access parameters without risking that 2 script libraries have an identically named function (e.g. getLocations() where one retrieves sales locations and the other manufacturing locations) that clash. The object name will help to distinguish them. Furthermore the call to the function is independent from the actual implementation. Doing a @DBLookup/@DBColumn for rather static values might not make a lot of sense, so some caching might make sense. Instead of building that into every page you can leave that to the function -- adding caching or reading config values differently will not change its signature (signature is the fancy way of saying: how it is called and what it returns). There is an additional reason why you want to stick to the "getParameterName()" syntax. It will allow you, if you want to, to replace your SSJS object with a managed bean that allows you more flexible options how to retrieve and cache your parameters. With a managed bean you also could use parameters directly in XPages' expression language (EL). You gain the option instead of writing #{JavaScript:companyConfig.getDepartments()} the more compact and understandable EL syntax #companyConfig.departments}. As said: only if you want to. Sticking to that syntax i SSJS doesn't have, as far as I can see, any substantial drawback. Planning for that keeps your options open.
As usual YMMV.


XPages - the "other" skills

In most of the XPages trainings I conducted in 2009 and 2010 there were participants with no prior exposure to Notes and Domino other than using it as a user. They all however had experience with web development. XPages' JavaScript, CSS and HTML was like their natural environment and they had no problem catching up with "the old Notes nuts". The later however, especially the "client only" developers struggled. They struggled not with DDE or XPages per se, but with the open standards it embraces. So it is back to school (unless you have a medical condition) for open standards. This is my cheat sheet for the different technologies: For all the books you actually might consider a Safari book subscription.
I know - it is a lot of stuff. On the upside: these are all mostly platform independent (OK, not the Java stuff, but that's at least vendor independent). And: you don't need to know all these to get started - it is rather the master's list.
As usual YMMV


XPages Medical Advice

The truth about XPages:
XPages can trigger Cainotophobia
But of course you know that and are in tune with the impermanence of being and continuous upgrade your skills.


Getting close with IBM OneUI

QuickImage Category
I recommended the use of IBM OneUI before including a helper stylesheet. While there is a complete 2.0 documentation (as well as the 2.1 version or v3.0 version) available, a little explanation about the how and why goes a long way. The OneUI is not so much about colours and fonts, but about structure, navigation and layout. Following its guidelines your applications get a consistent look, which user studies show increase user confidence about their ability to use your web application. This translates into less help desk calls. Any consistent framework does that, the advantage of the OneUI is that your applications start looking consistent to the IBM software in use. Furthermore when IBM releases a new visual style, your application can adopt it in no time. Lets have a look at the layout of a OneUI page. It consists of the following structure:
  • Global Navigation: Branding (corporate and application logo), links to core applications, usern ame, login and help
  • Application Navigation: Typically a tab bar on the left and the search box on the right. Search, while potentially acting on the current page content is considered an application level function. You could use a dropdown to allow the user to define the search context
  • Place / Page Navigation: Contains the page title on the left and a series of page level actions on the right. A page would be the place where you conclude an entire interaction. This is different from the Notes client or classic web application model where view (selection of information to act on) and document (the instance of information) are usually on different pages. Some rethinking required (Partial page refresh is your friend!)
  • Main Content: The main content area has one to three columns where the left and right column are optional. If the main content needs navigation or filtering that would be displayed in the left column with navigation above filters. In the right column contextual information can be displayed (e.g. in a Claim form the right column would show the account summary)
  • Footer: The footer area contains logo, copyright and related links that are not frequently uses (e.g. Feedback, documentation, other software)
Drawn on a wire frame you get this:

An empty page just containing the layout would look like this:
OneUI in green
This is also available in blue, red, gold, metal, black and white and it even is usable using Lynx. Branding fanatics will highlight that the logo area is tiny and the footer to long, however nobody stops you from altering the CSS, as long as you stick to the structure (the small logo saves valuable space above the fold). Some Lynx impressions:


Multi column checkboxes and radio buttons

Brian shared his implementation of multi-column check boxes. Having such a control mimicking the functionality of the Notes client is pretty helpful. Brian's approach however wasn't very re-usable, so I though of a way to come up with a more reusable solution:
Multi column check boxes in various views
The resulting custom controls can be easily added to any page with a few lines of code. You need to reference 2 supporting JavaScript libraries and then add a few lines to your page:
  1. < xc:MultiColumnCheckBox numberOfColumns="3"dataBindingName="document1.CheckDemo">
  2.     < xc:this.FieldValues>< ![CDATA[#{javascript:return ["black","blue","red","yellow","white","brown","gray","vomit","green"];}></xc:this.FieldValues>
  3.     < xc:this.disabledValues>< !CDATA[#{javascript:return ["gray","vomit"}]]></xc:this.disabledValues>
  4. </xc:MultiColumnCheckBox>
You have 4 parameters:
  • numberOfColumns: (Integer) number of columns to use
  • dataBindingName : (String) field to bind the control to including the datasource
  • FieldValues: (Array of Strings) Values to offer in the control
  • disabledValues: (Array of Strings - optional) Values that are disabled
The same parameters also work for a group of radio buttons:
  1. < xc:MultiColumnRadioButton numberOfColumns="3" dataBindingName="document1.RadioDemo">
  2.     < xc:this.FieldValues>< ![CDATA[#{javascript:return ["Porsche", "VW", "BMW", "Mercedes", "Opel", "Ford", "Audi","Lada"]}]]></xc:this.FieldValues>
  3.     < xc:this.disabledValues>< ![CDATA[#{javascript:return ["Lada"]}]]></xc:this.disabledValues>
  4. </xc:MultiColumnRadioButton>
The controls are available for download on OpenNTF (of course you use Guo Yi's Import and Export for Designer for this). Enjoy and share your ideas for v1.1
As usual YMMV.


Meet the XSP object

QuickImage Category
Look at the source code of any XPage in your browser (you are are using Firebug, are you?*) and you will find numerous invocations to the XSP object. This object sits on top of Dojo (at least for now) and is one of the reasons why type-ahead and partial refresh work so easily in XPages. Having a closer look at the object reveals a wealth of possibilities. A good starting point is an article by Teresa Monahan (one of our code goddesses in the IBM Dublin Software lab) on the Domino Designer WIKI. To get a deeper look simply examine the source. You can find the file in [Your Notes/Domino data directory]/domino/js/[DojoVersion]/ibm/xsp/widget/layout/XSPClientDojo.js.uncompressed.js. Let us have a look at some of the interesting functions and possibilities. XSP has a strong heritage in Dojo, but IBM might provide alternate implementations for specific needs in future. To get a head start in Dojo check out Mastering Dojo, Dojo: The Definitive Guide, Practical Dojo Projects or the other Dojo books. But back to XSP:
  • XSP.addOnLoad( listener )
    Adds a function "listener" to the code that is executed when loading a page.
  • XSP.attachClientScript(targetClientId, _event, clientScript), xsp.attachClientFunction(targetClientId, _event, clientSideScriptName)
    Attaches a function to a client element for a specific event. If the function exists use attachClientFunction. If the code is inside the variable "clientScript" use attachClientScript
  • XSP.setSubmitValue(value)
    Overwrites what a submission is sending back to the server. Use with caution!
  • XSP.canSubmit()
    Returns true/false. If a page has been recently submitted it returns false. Good way to prevent submit button double click hazards. To manually set it to true call XSP.allowSubmit()
  • XSP.addPreSubmitListener(formid, listener, clientId, scriptid)
    Adds a function (the listener) that runs before the form (formid) is submitted. Great way to run cleanup code in custom controls to make them self contained. Also a good way to have code run only for a specific button (clientid) rather than onsubmit in any case. The scriptid helps to check if the listener has been loaded already (e.g. on partial page refresh) to avoid duplicate loads. If listener returns false submission does not happen. Runs after the client side validators.
  • XSP.addQuerySubmitListener(formId, listener, clientId, scriptId)
    Same as above but runs before the validators
  • XSP.alert(message), XSP.error(message), XSP.confirm(question)
    Currently this are wrappers around alert() and dojo.confirm(). Using this functions has the advantage of instant beautifications once the XSP object gets upgraded
  • XSP.attachValidator(clientId, validator, converter)
    Allows to attach a validator to a client object. The better way to validate than writing code in the onSubmit event. Admittingly a little more complex. The function warrants its own blog entry (stay tuned)
  • XSP.validateAllFields(true|false)
    Default for this is false. When set to true all client side validation is executed not only all until the first error. You will get a number of prompts. Makes sense with an overwritten validation function, but that's a story for another time.
  • XSP.attachEvent(clientId, targetClientId, _event, clientSideScriptName, submit, valmode, execId)
    Attaches events to client side elements. The clientId is the element, the targetClientId the target for a partial refresh, _event the event to attach. More in a future blog entry.
  • XSP.fireEvent(evt, clientId, targetId, clientSideScriptName, submit, valmode, execId)
    Allows to trigger an event script.
  • XSP.attachPartial(clientId, targetId, execId, eventName, scriptName, valmode, refreshId, onStart, onComplete, onError)
    Attaches events to client side elements. The clientId is the element, the targetId the target for a partial refresh, eventName the event to attach. on(Start|Complete|Error) the functions to run locally around the ajax event. More in a future blog entry.
  • XSP.partialRefreshGet(refreshId, options), XSP.partialRefreshPost(refreshId, options)
    Triggers a partial refresh manually. For POST the refreshId must point to a form, for get it can be any element with a partial refresh defines. The options object is quite interesting to examine. Includes parameters and functions to execute in the life cycle.
  • XSP.attachDirtyListener(clientId)
    Listen to a specific field. If it changes consider the form "dirty". (Persil to the rescue). Use XSP.attachDirtyUnloadListener(message) to set a message. Typically you use that for "Your form has changed, do you really want to navigate away" type of messages.
  • XSP.trim(string), XSP.startsWith(string, seachString), XSP.endsWith(string, searchString)
    Convenience functions for string stuff.
  • XSP.serialize(object)
    Takes an JavaScript object and returns a textual representation. Currently JSON, but might change.
  • XSP.toJson(object), XSP.fromJson(string)
    Convert data from/to JSON format.
  • XSP.log(message)
    Send a message to the local log. Used for debug (mostly)
This is just an extract. I probably will cover usage details in future posts. You will have fun exploring more of it. As usual: YMMV

*Remark: Regardless what target environment environment you develop for: IE6, IE7, IE8, IE9, FF2, FF3, FF4, Opera, WebKit (The core of the browsers in iPhone, iTouch, iPad, Android, Chrome and Safari) or others, you do want to develop your XPages application using Firefox. The simple reasons are Firebug and Medusa.


Taming the categorized views - reloaded

QuickImage Category
My previous solution was designed for categorised views that don't use column totals. The difference between a view control with and without column totals is subtle but important. Without totals a category row is rendered with a colspan attribute that allows the categories to freely flow over the whole width of the table. Are column totals present no colspan is used and the column for the category must be wide enough to accommodate the full content. With a little JavaScript that can be fixed. The server side JS now looks like this:
  1. function getClientJSforTableLayout(tableID) {
  2.     /**
  3.       * This code has as result JavaScript source code that is send down to the client]
  4.     */
  5.     var curTable = getComponent(tableID);
  6.     var curID = getClientId(tableID);
  7.     var tablewidth = extractWidthFromStyle(curTable.viewStyle);
  8.     var result = "fixCatTableSpan(\""+curID;
  9.     result += "\");\n";
  10.     result += "fixCatTableWidth(\""; // This holds the JavaScript we return to the browser
  11.     result += curID + "\",\"";
  12.     // Here we compute the sizeString
  14.     var kids = curTable.getChildren();
  15.     var defaultPercent = Math.floor(100/kids.size())+"% "; // You have to love flexi data types
  16.     var perCentArray = [];
  17.     for (var x in kids) {
  18.         var styleCandidate =;
  19.         if (styleCandidate == null) {
  20.             perCentArray[perCentArray.length] = defaultPercent;
  21.         } else {
  22.             var candidate2 = extractWidthFromStyle(styleCandidate);
  23.             if (candidate2 == "") {
  24.                 perCentArray[perCentArray.length] = defaultPercent;
  25.             } else {
  26.                 perCentArray[perCentArray.length] = candidate2;
  27.             }
  28.         }  
  30.     }
  31.     result += perCentArray.join("\", \"");
  32.     result += "\",\""+tablewidth+"\");";
  33.     return result;
  34. }
  36. function extractWidthFromStyle(styleCandidate) {
  37.     var result = "";
  38.     var whereisWidth = styleCandidate.indexOf("width:");
  39.     if (whereisWidth < 0) {
  40.         result = "";
  41.     } else {
  42.         var workString = styleCandidate.substr(whereisWidth+6);
  43.         var hasSemiColon = workString.indexOf(";");
  44.             if (hasSemiColon < 0) {
  45.                 result = workString;
  46.             } else {
  47.                 result = workString.substr(0,hasSemiColon);
  48.             }
  49.     }
  50.     return result;         
  51. }


Developing your XPages skills

QuickImage Category
When a body of knowledge reaches a certain size specialisation kicks in. So while we all can try to be like Leibniz or Da Vinci at the end some specialisation keeps us sane. With the growing body of knowledge in XPages you need to think where you want to go deep unless of course you are a Jack of all trades
XPages skills go in all directions
There are different directions to go: front-end or back-end, business applications or common components.
  • When you look at the back-end you want to understand the Domino server and storage in detail. You will encounter XPages' JSF underpinning and manage quite some beans. You will boldly go where "no LotusScript programmer has gone before". That is an area where knowledge of Java is a must. You might link XPages to host systems or RDBMS taking advantage of all the Java libraries at your hand
  • Deep inside the back-end there is the OSGI extension model. You write your own components to extend the available capabilities. Most likely you don't write them for a single purpose but for general reuse
  • On the front-end you want to understand JavaScript and the Dojo Toolkit. When you build shared components and controls you design them to be parameterised for easy reuse. The Dojo types you create can simply be used by your business developers in a declarative way. You have a good understanding of the IBM OneUI and reasonable CSS skills
  • You are the UI guy. You know CSS inside out, as well as all the Browser quirks. You design new Themes
All these specialisation types were almost non-existent in classic Notes development and you actually can get away without them still building great Domino applications on XPages. You are the Lotus Domino Business application developer (a.k.a the Classic): You work with what is given. Only now the group of givers has grown (see above). You work with script and are OK to switch from LotusScript to JavaScript. The only thing you care about Dojo is to know the name of the controls to put in the XPages control properties. You are perfectly happy using the OneUI. AND: You understand your users (you always have), prototype applications rapidly and create happy users before morning tea break. That was what Notes and Domino development are about: rapid user satisfaction.


Taming the jumpy categorized Views in XPages

QuickImage Category
I am not a fan of categorized views in web applications and have suggested alternative uses before. However beautiy is in the eye of the beholder, so categorized views are used. When a category is rendered it sits on its own tablerow (tr) in a tabletag (td) that has a colspan attribute. This effectively disables the column width settings for your view until you expand the category to show a full row of data resulting in some "screen jumping". While that won't sink the Titanic it is a little odd. Short of rolling your own rendering using a repeat control, there is a fix consisting of 2 JavaScript functions (one for the server, one for the client and a little addition in the source code panel. This works in all categorized views. However when you have a column with a column total rendering might be different since XPages doesn't render the colspan attribute then. Lets get started. Open your source pane and locate the view panel and the xp:this.facets. Here you insert a scriptBlock like this:
  1. {xp:this.facets}
  2.     {xp:pager partialRefresh="true" layout="Previous Group Next"
  3.     xp:key="headerPager" id="pager1"}
  4.     {/xp:pager}
  5.      {xp:scriptBlock xp:key="south" id="scriptBlock1"}
  6.     {xp:this.value}{![CDATA[#{javascript:return getClientJSforTableLayout("viewPanel1");} ] ] }{/xp:this.value}
  7.      {/xp:scriptBlock}
  8. {/xp:this.facets}
(Replace { } with < and > except inside the CDATA section). Then you add 2 JavaScript libraries to your XPage: CatViewPatches.js and TableSupportFunctions.jss. They will fix the column width. One caveat: you have to specify the width for all columns otherwise the unspecified columns get 100/{number-of-columns}% width assigned.


Partial Page Rendering in XPages

QuickImage Category
More goodness coming along in XPages 8.5.2 (Disclaimer: it is still Beta ...) An XPage is a collection, or to be precise, a tree of controls with an xp:view tag as root control. The xp:view tag translates into a instance of the UIViewRootEx2 class. In theory you could subclass this but then you stray far away from the path of IBM support. In some case (think Ajax updates) it would be nice if you could request the render result of just one control (and all its children of course). In 852 this is possible using the ?$$axtarget= directive. It takes the client id as parameter and requires a keyword (you can use to identify the action) between page name and directive. So your URL could look like this:
The control doesn't need to check for the value of [SomeactionName], it is up to you if you want to use that. A sample function giving you the url for a given command and control could look like this (you want to add error handling for production use):
  1. function getSpecialAction(actionName, componentName) {
  2.     var toRenderPartial = getComponent(componentName);
  3.     return 'myPage.xsp/'+actionName+'?$$axtarget='+toRenderPartial.getClientId(facesContext);
  4. }
As usual YMMV


Attachment URLs in XPages

QuickImage Category
In classic Domino applications directly addressing attachments is well known and simple. Your URL has the format:
http(s)://[yourserver]/[application.nsf]/[viewname|0]/[UNID| ViewKey]/$File/[AttachmentName]?Open
While this will continue to work on the Domino server for the forseeable future you want to learn about the XPages syntax. The very obvious reason: the classic Domino URL doesn't work in the Notes client (XPiNC) and you want to make your applications work cross-platform don't you? Also we don't know what additional options handling attachments via XPages might bring in the future. the syntax is a litte more complex.:
http(s)://[yourserver]/[application.nsf] /xsp/.ibmmodres/domino/OpenAttachment/ [application.nsf]/[UNID|/$File/[AttachmentName]?Open
A few remarks:
  • The URL contains the application location twice. From my tests it looks like the /__[ReplicaID].nsf/ doesn't work for here. I haven't tested what happens when you specify 2 different applications in the front and the back
  • Since there is no view involved in the URL addressing attachments using an attachment key won't work. You have to use the UNID. This requires extra care when you build applications where the attachment URL is supposed to be shared (like send in a chat or an eMail). Deleting a document and recreating with an updated attachment will alter the URL. So ou want to cater for that case with a specific XPage rather than a raw URL.
  • When you attachment is stored inside a RT field (e.g. using the upload control where you target a field) you can replace "/$File/" with the field name e.g. "/Body/"
To make my life easier I use 2 functions so I get an attachment URL with a single call:


XPages training coming up - in ANZ

Dates and locations updated!:
The release of Domino 8.5.2 draws closer and I'm happy to announce more training events (hosted by your's truly):
  1. Beginners XPages: 10 Aug 2010, Auckland, New Zealand
  2. Advanced XPages: 11 Aug 2010, Auckland, New Zealand
  3. Beginners XPages: 12 Aug 2010, Wellington, New Zealand
  4. Advanced XPages: 13 Aug 2010, Wellington, New Zealand
  5. Beginners XPages: 17 Aug 2010, Sydney, Australia
  6. Advanced XPages: 18 Aug 2010, Sydney, Australia
  7. Beginners XPages: 19 Aug 2010, Melbourne, Australia
  8. XPages for the Brisbane Lotus User Group: 20 Aug 2010, Brisbane, Australia


IBM OneUI v2

In Domino 8.5.1 the IBM OneUI v2.01 files were sneaked in. In Domino 8.5.2 they will be official. (I've written about IBM OneUI before). So it is time to look at the official 2.0 documentation (or the 2.1 version) and make yourself familiar with the concepts. To use the OneUI ou just manually type oneuiv2 as your theme and your application will pick it up. Make sure that you type that all small, so it works on all servers. On my last count the styles contained 287 unique .lotusXXX class names and 102 .xspXXX class names in various combinations of HTML elements and nesting orders. So somebody spend a substantial amount of time to get all the various cases covered. I probably will blog about the structure a little more in future. To make use of these classes more convenient I updated my dummy stylesheet and provide 2 stylesheets, one for .lotusXXX and one for .xspXXX to include in our projects. There are no CSS definitions inside, so you won't have unwanted side effects.


XPages and Form Fomulas

One of the unique features in Notes and Domino are form formulas. On a view you can specify a formula that alters the form used to show a form. This is very handy to show the same document reformatted for different stake holders, so they can focus on what is important to them
Form Formulas in Notes Views
Form formulas are a tried and tested solution. So I was wondering what happens when you bring XPages into the mix.


Rendering ODF documents in your XPages

In Tim's and my Lotusphere session AD111 we rendered data from an XPage into a ODT document. Once you get the hang of it and it is easy to implement export to text, spreadsheet or presentations. One nice aspect: no server side OpenOffice or Symphony code is needed. Here is how you do it:
  1. In DDE use Window, Show Eclipse Views, Other (Alt+Shift+Q Q) and select "Navigator"
  2. Locate the WebContent/WEB-INF folder and add 2 directories: src and lib
  3. Download the Java toolkit from ODF Toolkit and add odfdom.jar to your WebContent/WEB-INF/lib directory
  4. Right click on your NSF and select Project Properties. Then select Java Build Path. In the source tab add the src folder you created in step 2. In the Libraries tab add the odfdom.jar you copied in step 3
  5. Create the class you want to use for rendering the content (rendering logic goes there). It should have a method that takes an OutputStream as parameter (I called it render)
  6. Optional: define your new class to be a managed bean. Once you do that it can be used as if it is a native global variable. To do this locate the faces-config.xml in the WEB-INF directory and add a bean definition. Could look like this:
  7. Create a new XPage, go to the page properties, basics and set rendered=false. You can see in the XSP source
    <xp:view xmlns:xp="" rendered="false">
  8. Add code (see below) in the beforeRenderResponse event. It is important to use this event, so you can get hand on the OutputStream rather than the writer
  9. The ODFToolkit requires one additional setting for security, so locate [NotesInstallDir]/jvm/lib/security/java.policy file (you need to do that on the server too) and add:
    grant { permission java.lang.RuntimePermission "shutdownHooks"; };
  10. You are all set. Enjoy. As usual YMMV


Peek-a-boo into the XPages classes

XPages and server side JavaScript come quite some classes (and I'm not counting the Java classes you could use). We have:
  • Class Library: DOM
    • DOMAttr
    • DOMCDATASection
    • DOMCharacterData
    • DOMComment
    • DOMDocument
    • DOMDocumentFragment
    • DOMDocumentType
    • DOMElement
    • DOMEntity
    • DOMEntityReference
    • DOMException
    • DOMImplementation
    • DOMNamedNodeMap
    • DOMNode
    • DOMNodeList
    • DOMNotation
    • DOMProcessingInstruction
    • DOMText
    • DOMUtil
    • NamespaceContext
    • NamespaceContextImpl
  • Class Library: Domino
    • DirectoryNavigator
    • NotesACL
    • NotesACLEntry
    • NotesAdministrationProcess
    • NotesAgent
    • NotesAgentContext
    • NotesBase (recycle() is defined here)
    • NotesColorObject
    • NotesDatabase
    • NotesDateRange
    • NotesDateTime
    • NotesDbDirectory
    • NotesDirectory
    • NotesDocument
    • NotesDocumentCollection
    • NotesDxlExporter
    • NotesDxlImporter
    • NotesEmbeddedObject
    • NotesError
    • NotesException
    • NotesFactory
    • NotesForm
    • NotesInternational
    • NotesItem
    • NotesLog
    • NotesMIMEEntity
    • NotesMIMEHeader
    • NotesName
    • NotesNewsletter
    • NotesNoteCollection
    • NotesOutline
    • NotesOutlineEntry
    • NotesProperty (used in Mashups and composite apps)
    • NotesPropertyBroker
    • NotesRegistration
    • NotesReplication
    • NotesReplicationEntry
    • NotesRichTextDoclink
    • NotesRichTextItem
    • NotesRichTextNavigator
    • NotesRichTextParagraphStyle
    • NotesRichTextRange
    • NotesRichTextSection
    • NotesRichTextStyle
    • NotesRichTextTab
    • NotesRichTextTable
    • NotesSession
    • NotesStream
    • NotesView
    • NotesViewColumn
    • NotesViewEntry
    • NotesViewEntryCollection
    • NotesViewNavigator
    • NotesXSLTResultTarget
  • Class Library: Runtime
    • I18n
    • Locale
    • TimeZone
  • Class Library: Standard
    • Array
    • Boolean
    • Date
    • Function
    • Math
    • Number
    • Object
    • RegExp
    • String
  • Class Library: XSP
    • DirectoryUser
    • NotesXspDocument
    • NotesXspViewEntry
    • XSPContext
    • XSPUrl
    • XSPUserAgent
  • Function library: @Functions (with many functions- for the next time)
  • Function library: XSP Functions (with many functions- for the next time)
That's it. Read the full list with all properties and methods.


XPages View Pager - Advanced edition

XPages does a fe nice things automagically for you. Thinks like giving you the content of a view page by page. While this is impressive on the first view a lot of the potential users would like to see more options in that navigation/paging control. This got me thinking and I drafted how the "Ultimate View Navigator" would look like. After all Bob Obringer designed his classic Domino navigator partly based on my suggestions (Unfortunately Bob's blog is gone). So I doodled around with my favourite screen design tool and here is what I came up with:
Advanced View Pager control for XPages
I think it covers all the usual bases. A few caveats: It hasn't been designed for categorised views, but will display single category selections well (for my view on categorised views, check my earlier post). It also will not perform nicely when you use reader fields protected documents (and again that topic has been covered before). Did I miss anything in the UI?


Software Design and Components in XPages

In Domino 8.5.1 we will have a new Design Element "Component". This design element can be used in Portal, Mashups and the Notes sidebar as iWidget as well as composite component in the Notes client. In a component you pick 3 XPages (could be the same for all 3 cases) for read, edit and help mode. You also define the events the components "knows" and what type of event is is for that component: sending, receiving or both. An event is associated with a data type that can be primitive (String, number), JSON or a complex data type you defined with that component.
Components are the Lego blocks of your applications. A system designer now can quickly build "empty" components wire them up and then leave the implementation to the XPages development team. This could accelerate Domino's RAD capabilities further.


Showing Categorized Views in Web Applications

QuickImage Category  
Categorized views are kind of a trademark of Lotus Notes (Client) applications. We like them, we build them, we love them. We also want to see them on the web. There is only one small issue: That display of information is pretty unique to Notes. You do find this tree/table combination in other applications only to display files (like Nautilus, Finder or KDE) but not data. So a categorized view is kind of odd on the web. I played around to find alternate displays for single and multiple category data. Here is what I came up with.
  • Single Category view with Listbox
    The category isn't part of the view itself but a picklist on the left (which might be filled by a @DbColumn). The table shows the current selection matching the selection. Works out of the box already today. Interesting extension challenge: allow selection of multiple entries in the listbox.
    Single Category View with Picklist
  • Single Category view with Combobox
    A variation of the first theme. Useful if you have a lot of view columns and need the real estate on the left. Variation: Instead of the dropdown: show a link that uses a popup to allow selection of the category to show.
    Single Category View with Dropdown combo
  • Multiple Categories with Combobox
    This actually works also with a sorted view since the limit key takes an array as entry. As variation similar to the previous example could be to show a breadcrumb link list that allow to click or hover to show the selection. Challenge: How to make it obvious that you need to select/narrow from left to right. Extra challenge: when you change box 2 and the value in box 3 is no longer an available value
    Multiple Categories with Combobox
  • Multiple Categories in tree/table combination
    Looks like a file dialogue, so users should be familiar. You trade horizontal space for vertical space. Makes navigation in categories more accessible since all categories are available any time
    Multiple Categories in tree/table combination
  • Pivot view on 3 categories
    Category 2 becomes the columns of the table, Category 3 the rows. Adds sums to rows and columns. Good base material for graphs. Challenge: decide on the aggregation mode: sum/average/percentage -or- optional display of such a column
    Pivot view on 3 categories
  • Pivot view on 3 categories with data rows
    Similar concept like the previous but with individual data rows displayed. Might show additional aggregation rows or columns
    Pivot view on 3 categories with data rows
Now someone needs to build all these custom controls. All the images have been build using Balsamip Mockups


More XPages Training coming your way

We have scheduled 3 more XPages training classes in AP:
  • 28/29 July, Jakarta - Indonesia (contact Gunawan for details
  • 31Aug/01Sep, Seoul - Korea (contact Kim Ky Young for details)
  • 22/23 Sep, Hanoi - Vietnam (contact Nguyen Hoai Nam for details)
And yes. Class delivery is by yours truly.


XPages Custom Controls, what will you use?

QuickImage Category
XPages is a constant topic when talking to Domino developers. One of my favorite questions to them is: "What (custom) controls are you expecting to use". The controls might exist, being build or being acquired. This is the unsorted list of what I heard so far:
  • Names fields with address picker
  • Custom address dialogs
  • Numeric spinners
  • Graphic controls connected to a data source
  • Custom data sources control
  • Controls supporting encryption/decryption
  • Workflow controls
  • Tagging input/Tag cloud from Lotus Connections
  • Controls that are Sametime aware
  • Chat controls
  • Data filter control that looks like the Excel filter mechanism
  • Tree control to render an outline
  • Dialogbox control
  • Login/User account control
  • Pivot table linked to a data source
  • Picture gallery controls in various forms and shapes including cover flow
  • Task list controls (add this to a task in Notes or Activities)
  • Notification control (send a message to notify others: eMail, Chat, Twitter)
  • Document history controls
  • Access control control (reader/author field management)
  • Translation control (translate the DATA)
  • Rating control
  • Feedback control
  • Shopping cart control
  • Lotus Connections Business Card control
  • Attachment control. Here I got multiple ideas around attachment management including Quickr integration
  • "See also" control: add a link to the page
  • Application Help control
  • Comment control
  • Expiry control (from just adding a date to sophisticated lookups in corporate policies databases)
  • Media view control (Flash, Media players etc.)
  • Inline edit control (edit one field without switching to edit mode)
  • Spreadsheet control (with Export - thx John)
  • Search control (from search in a view to Enterprise Search to Google search)
  • Poll control (ask a question and render the results)
  • ToolTip control
  • "Print This" control
  • PDF control
  • Gannt chart control
  • Scheduling control (using Quarz) for tasks
  • Balsamiq Mockup control
  • iLog Rule engine control
  • Update: New control requests/ideas
    • Query Building control (Formula/FullText), thx Erik
    • Datasource merge control: Have 2 identical structured datasources and merge their content into a single grid/table
    • Login/Credential for SSO, thx Kevin
    • Transformation/Export control: transform the content of the pages into other formats. A superset of PDF/Spreadsheet export.
    • Paypal control: there are new more open APIs available now
    • Actionbar / Dropdown menu control
    • Right Mouseclick menu
    • Document lock/unlock control
    • Timer control: execute JavaScript and/or Ajax calls on time
    • Slilder control for views, data tables and repeat control
Interesting insight: We are seeing "small" controls like the names field as well as "big" ones like an entire Workflow piece. To be clear: that's what customers and partner said, that's *not* taken from any IBM ToDo list (or at least none I'm aware of). Some of these controls are already in the wild.
What controls are you looking to use in your application?


Building a fieldset custom control

HTML has widely used and more exotic tags. Fieldset and Legend being of the later type. Theo reminded me, that they can be useful and asked how to incorporate them into a XPage. The best approach is to create a custom control with custom parameters and an editable area inside and simply type the html controls into the source code. The legend is rendered by a computed field, so translation will be able to pick it up.
Custom Control containing a FieldSet
The source code of the XPage looks like this (note: the custom property doesn't show up here since it is stored in a file hidden from the Domino Designer perspective):
<xp:view xmlns:xp="">
<xp:text escape="false" id="legendText" value="#{javascript:compositeData.legendText}">
<xp:callback facetName="facetFieldSetContent" id="callbackFieldControlSet" />

When using the custom control in a XPage or another control it looks like this (note the content inside the "this.facets") is what you put in there. can be a panel or table with fields in it or another control.
<xp:view xmlns:xp="" xmlns:xc="">
<xc:FieldSetControl id="MyFieldSet">
<xc:this.legendText>This is the &lt;b&gt;legend&lt;/b&gt; text</xc:this.legendText>
<xp:panel xp:key="facetFieldSetContent" id="stuffInsideTheFieldSet">
<xp:label id="label1" for="inputText1" value="Test Field">
<xp:inputText id="inputText1" />

As usual: YMMV


No more DIV tag hacks in XPages to get your favorite Dojo Widget

Domino 8.5.1 went into controlled beta recently. While others do performance testing provide us with and performance tips I poked around detail improvements. One of the early tips to integrate dojo components into XPages is to use a DIV wrapper. In 8.5.1 this is no longer necessary, since you now can specify dojoType and dojoAttributes directly on a panel (and some other custom controls). A nice step to cleaner code
Dojo Properties in Domino Designer 8.5.1


XPages Workshops in XI'AN (today) and Hong Kong (06/07 July 2009)

Today I completed a short introduction into XPages for a business partner in Xi'an. Initially the participants were a little stiff:
Famous people from Xi'an
But with food and demo (click on the picture above) they warmed up to the new way of doing things. I'll spend the weekend in Xi'an visiting one of the five holy mountains of the Taoist: Mount Huashan. I'm curious how this will work out.
I'm pleased to announce, that I will conduct another XPages workshop, this time in Hong Kong. (Update: Date for HK has changed!)The class will be on the 06/07 July 2009 and is open for registration. For details, location, fees etc. please contact Tony KT Lee from IBM Hong Kong.


Using the IBM OneUI in your XPages applications - now with CSS class picker

In his outstanding XPages tutorial Declan Lynch used IBM's OneUI for designing the layout and colors. This is very smart since it releaves you from thinking about all this and allows you to focus on the functionality. If you haven't worked through that tutorial, go there now and have a look. One thing Declan had a hard time with is to figure out what exactly the CSS classes are that the OneUI makes available - He did a great job here - (and then what to use where) since the CSS style picker doesn't show the classes available in the Themes, but only in the CSS you added to the page directly.
I thought that's not that nice and needs fixing. And here you go: Import the OneUIDummy.css into your database and add it to your pages. The file contains all classes used in the OneUI version on the Domino 8.5 server but no style definitions by its own. You now can pick the classes from the style picker and actually use the definitions in the server-side OneUI Theme. A little step for a developer and a big step for convenience. (You still need to learn what class to use where).
As usual: YMMV.


XPages advanced challenges (at least for me)

I pride myself with being able to help students quickly over their stumbling blocks. But is made me stumble:
Domino Designer for Eclipse Traditional Chinese
A good spatial memory and an aptitude for icons helps a lot. We are definitely having fun here.


XPages Workshop in Taipei

The XPages workshop in Taipei is well under way. We have 32 participants at the IBM Education center happily coding away on Domino Designer 8.5. The joy of travel is to experience all the little cultural differences. Like the coffee bar:
What type of tea would you like?
Only one of the cans actually contains coffee. The other 3 (x2) cans offer you different sorts of tea freshly brewed. I like that much better than the tea bags you find in other locations. Maybe that's the reason why the class is so fully focused.
A full classroom of future XPages developers
There will be more workshops in Taiwan, Cory Ma knows the details.


XPages Workshop Impressions from Bangkok

The XPages workshop proved to be very popular. About 30 participants showed up to get their take on web2.0 development with Domino Designer 8.5. Impressions from earlier workshops got confirmed:
  • Everybody loves XPages
  • Developer with a background in web development grasp XPages faster than classical Notes developers
  • Exercise 17 is the most difficult one, but everybody can complete it
  • The XPages outline (in the default Domino Designer perspective on the lower left) is a huge time saver navigating your XPage
  • Having the ability to inspect the source code can accelerate trouble shooting tremendously
  • A modal window to enter code is a nuisance
For business partners the workshop run as "bring your own laptop". To my surprise quite a number had Windows ME 2009 Edition Vista on their machines and quite some trouble getting the local preview of Domino designer to work. This is a known problem that plagued others too but has been solved one way or another. Just keep in mind: Never ever install a data directory inside the place where programs go (I think we need to adjust our installer defaults) and be clear what runs/blocks Port 80. Look out for Skype (can be switched off in the settings), Firewalls, Webservers etc.


XPages Custom controls parameter definitions

One of the very cool features in XPages is the ability to create your own custom controls. While on the first look one could say "well, they are just subforms with a new name" the second look shows bigger differences:
  • Custom controls can appear more than once on a page, subforms only once (yes there was the possibility to add a subform twice if it didn't contain any fields, but that was not intended)
  • Custom controls are self aware: they "know" what is inside and what is outside
  • Custom controls have parameters. You can define parameters that the custom control will address as CompositeData.[parameterName] and the hosting pages address as [controlName].[parameterName]
When you look at the source code of a page you can clearly see the parameters you might have set for a custom control. However when you look at the source code of your custom control, you won't see any definition. Puzzled me for a moment, so I asked the master. At the very end XPages is a JFS application. In JFS a custom control has 2 files (at least): the code and the definition. XPages shields you from the complexity by allowing you to edit everything in one UI. But if you *really* want to have a look, add the Navigator window to your Domino Designer perspective and navigate to the custom control directory. Here you will find MyControl.xsp as well as MyControl.xsp-config. If you want to understand what the containing XML is all about, take a peek at a JSF tutorial


Upcoming XPages workshops in AP

We have new dates for XPages workshops: Places are limited, so be quick.


XPages workshop in Kuala Lumpur and Manila

Last week I conducted our famous XPages workshop in Kuala Lumpur and this week in Manila. I found a high number of encouraging pointers:
  • participants generally love XPages
  • In both classes more than half of the participants had no or less than 2 years experience with Lotus Notes and did quite well
  • One customer shared: "We haven't developed new Domino applications for a few years now, but with XPages we will resume creating new applications for the Lotus Domino server"
  • Developers with exposure to web development snap XPages up easily, old Notes hands need to unlearn a few things first
  • The repeat control is everybody's favourite measured by the time they spend with that exercise (repeat after me: with great powers comes great responsibility)
Next stop: Melbourne, Australia 29/30 April 2009.


XPages and Validation

QuickImage Category  
In our XPages workshop we have a number of exercises that deal with validation. XPages allows you 4 different approaches when and how to validate (not counting the on-field-exit validation which is a very bad habit anyway - never validate on exit, just hint, that this will not submit):
When you look at applications you typically find validation code in the submit button or attached to the submit event (that would be the second from left). While that is wide spread it is not the best way to do that in XPages. Looking at the possibilities we have 2 discussions: server vs. client, validator vs. Button/Event code. You have to be clear about the execution sequence: when a validator fails the submission code isn't executed. If the client validation (by any means) fails no server validation takes place.
  • Client side validation has the clear advantage to give faster feedback, since no round-trip to the server is required. However it only can be used as a measure for user convenience since it can be canceled out using the appropriate tools (Firebug anyone) easily. Client side validation needs all information for the validation downloaded into the client. So you are limited to simple validations like required fields, consistency checks, data types etc. Anything that requires server information (e.g. Is there budget left, is this the right approver) better lives on the server side. Also currently the client side validation in XPages relies on a simple alert() to notify of validation failures and users are fast as lightning to click away prompts without reading them.
  • Server side validation requires a round trip to the server (full page or a ajax call), so it will take longer. On the server side you can rely on any resources or lookups to protect data integrity. Also server side validation is less prone to manipulation (you would need access to the server sources). In XPages you also have better UI capabilities.
In conclusion: Client side is for convenience, server side for the "real stuff".
  • Validation in code (a button, the submit event etc.) is a typical way validation is done. Being the prevalent way doesn't make it right <g>. You need to roll your own notification mechanism (like updating a label) and tend to tie your validation into the UI. Also when you remove a field the validation routine is likely to break. Last not least: you have a hard time documenting what gets validated and why. (You see where I'm going with that)
  • Validators are defined together with a field and open a series of posibilities. XPages offers 9 different validators.

    You can write JavaScript, regular expressions, check for data types or roll your very own. All you can do in a button/event code you can do in a validator. Since the validators themselves don't interact with the UI the designer can decide how to surface the messages without changes to the validation code. When you remove a field all its validation code goes with it, so maintenance gets much easier. Last not least: you can run an XSLT report against your XPages source and render a report that shows a field with all the defined validators, which makes documentation easier.
  • Form Validation are the @Formulas defined in your classic Notes form. They only fire when you have specified "Run form validation" as "On Save" or "Both". Typically you would use those when upgrading existing applications.
In conclusion: Validators are the way to go. Read more about it in the Domino Designer WIKI on IBM DeveloperWorks. One important aspect to consider when you start and mix the validation methods is the execution sequence: From left to right: the first validation that fails will stop the process. So if (2) fails (3) - (5) never get executed!


Overheard at a developer discussion

QuickImage Category    
XPages feedback from an early adopter:"I'm now able to call our code from an XPage... and equivalent transactions that were taking a full second (or more) to complete in a Domino agent are now completing in 20 - 40 milliseconds. And we haven't even optimized it yet; that 50x performance boost is just from running it in XPages instead of in agents."
What do you want more?


XPages Workskops coming up.

Tim sums it up:
So it seems that word is spreading about this great XPages stuff, (so someone does read my blog) and I am being asked what the plans are to deliver it in other countries. January is going to be planning month and Lotusphere 2009 So here is who to contact if you want to add you name to the list of people/countries that are interested.
to find out how to contact them you can use the WhoIs application.
Only Stephan and I know about this blog post so far so when you contact the others, please be gentle. ;o)
  • EMEA: Tim Clark Please leave comments on this post to show an interest for EMEA.
  • Asia Pacific: Qiao Ming Chen for China, Stephan Wissel (me) for Asian & India, Atsushi Sato for Japan
  • ANZ: Craig Hart for Australia / New Zealand
  • Americas: Craig Wolpert


XAgents - Web Agents XPages style

QuickImage Category  
In Domino Agents are the Swiss Army Knife of development. They can be called from the client, triggered on schedule, automatically run on an event, be associated with a web action (Open, Close) or directly called from an URL using ?OpenAgent. The ?OpenAgent use of agents is subject of this blog entry. A agent triggered from an URL can use Print statements to output results back to the browser. When the first line of your print statements is "Content-Type:..." you also can serve XML or JSON or whatever exotic format your agent logic was able to conceive. Agents come with the usual drawback: the runtime environment is intitialized new for every agent run, so you need to update a document if you want to keep track (Replication conflicts anyone?)
XPages don't have a notion of agents (you can call them in the backend, but you won't get anything back from the print statements), so it seems a step back. On closer inspection however one can see, that XPages offer a greater flexibility for the same pattern of use. Every element on an XPages has a "render" attribute which determines if the element should be rendered. "Every" includes the page itself. So when you set the page's render property to false, well the XPage engine renders nothing. So you get an empty canvas you can paint on. XPages still provides you with the page event and access to scoped variables, so you can code better performing "agents" since you can keep lookups in memory. To get the equivalent to the LotusScript print you need the ResponseWriter from the externalContext. To set the content type you use the response.setContentType also to be found in the external Context. A sample snippet you could put in the afterRenderResponse event of your page would look like this (error handling omitted):
// The external context gives access to the servlet environment
var exCon = facesContext.getExternalContext(); 

// The writer is the closest you get to a PRINT statement
// If you need to output binary data, use the stream instead
var writer = facesContext.getResponseWriter();

// The servlet's response, check the J2EE documentation what you can do
var response = exCon.getResponse();

// In this example we want to deliver xml and make sure it doesn't get cached
response.setHeader("Cache-Control", "no-cache");

// Here all your output will be written
writer.write("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<christmas date\"24Dec\" />");

// We tell the writer we are through
// Update:On 8.5.2 or later writer.close() seems not be necessary/be rather harmful. Read the revisited entry
More information about the FacesContext can be found in the Domino Developer Wiki.

Bonus Tip: Since you do not render anything from the XPage you just used, why not take that empty page and write the documentation for that XAgent?
Update: Other posts on rendering your own output in XPages: Have fun writing XAgents!


XPages architecture from a web perspective

Yesterday I mused about how you can add a Domino 8.5 xPages server to an existing Domain without changing the existing infrastructure. Looking at the picture I realized, that while its function seems to be obvious, it actually might only be obvious to the initiated. So I went back to the drawing board. In a typical web application setting using other technologies (like JSP, ASP or PHP) you typically have a web server, an application server and a database server (they can and do sit quite often on the same physical machine). This is well understood by almost all IT decision makers. So for a Notes application web enablement you need a web server. This would look like this:
You add a "Web Frontend Server" to your existing infrastructure. It is like adding an Apache to an Oracle database or MS-IIS to MS-SQL. The nice thing in the Domino stack however is, that you don't need to acquire new skills to administrate this "Web Frontend Server". The box is simply your's truly Domino 8.5


The multi-faced nature of the NSF

QuickImage Category  
I'm running the XPages enablement workshop in Beijing this week. Having to teach other about new functionality guarantees either dispair or insight. Today I had a rather insightful moment. Very often the NSF gets blasted for being a "flat-file-non-relational-whatever" file. When having a closer look at the format however you will realize, that it is a quite amazing part of technology:
  • Build to function as a "whatever-you-want-to-dump-into" data store
  • Build backward compatible. I still can open R2 databases in my R8.5 client
  • Build to run on a client and on a server with high concurrent access
  • Build to be robust: event if you tourture it (diskspace, sectors etc.) it will mostly recover
  • Build to provide fulltext search access
  • Build to be what you want it to be: if you happen to be a Notes client or a Domino server, it is a repository for design elements and a document database. If you happen to be the Eclipse IDE (or a webDAV client) it is a file system. If you happen to be the XPages server task it is a WAR application store. And if you happen to be David Allen, it is your trusted system for Getting-Things-Done


This site is in no way affiliated, endorsed, sanctioned, supported, nor enlightened by Lotus Software nor IBM Corporation. I may be an employee, but the opinions, theories, facts, etc. presented here are my own and are in now way given in any official capacity. In short, these are my words and this is my site, not IBM's - and don't even begin to think otherwise. (Disclaimer shamelessly plugged from Rocky Oliver)
© 2003 - 2017 Stephan H. Wissel - some rights reserved as listed here: Creative Commons License
Unless otherwise labeled by its originating author, the content found on this site is made available under the terms of an Attribution/NonCommercial/ShareAlike Creative Commons License, with the exception that no rights are granted -- since they are not mine to grant -- in any logo, graphic design, trademarks or trade names of any type. Code samples and code downloads on this site are, unless otherwise labeled, made available under an Apache 2.0 license. Other license models are available on written request and written confirmation.