Search

Domino Upgrade

VersionSupport end
5.0
6.0
6.5
7.0
8.0
8.5
Upgrade to 9.x now!
(see the full Lotus lifcyle) To make your upgrade a success use the Upgrade Cheat Sheet.
Contemplating to replace Notes? You have to read this! (also available on Slideshare)

Languages

Other languages on request.

Twitter

Useful Tools

Get Firefox
Use OpenDNS
The support for Windows XP has come to an end . Time to consider an alternative to move on.

About Me

I am the "IBM Collaboration & Productivity Advisor" for IBM Asia Pacific. I'm based in Singapore.
Reach out to me via:
Follow notessensei on Twitter
(posts)
Skype
Sametime
IBM
Facebook
LinkedIn
XING
Amazon Store
Amazon Kindle
NotesSensei's Spreadshirt shop
profile for stwissel on Stack Exchange, a network of free, community-driven Q&A sites

02/04/2017

Project Deep Purple - IBM Notes Native for iOS

Category

We all heard the announcements around Project Pink headed by Jason R. Gary, the future of IBM Connections. Attending the conference we could admire him all clad out in pink.

However there is something more going on (and I'm not talking about his "don style" haircut). Besides the pink suit, Jason has been spotted in a deep purple jacket on several occasions, like his FossASIA talk. Digging deeper it seems IBM collaboration projects are now colour coded. Purple is chosen quite deliberately: The color purple is a rare occurring color in nature and as a result is often seen as having sacred meaning. Purple combines the calm stability of blue and the fierce energy of red. The color purple is often associated with royalty, nobility, luxury, power, and ambition. Purple also represents meanings of wealth, extravagance, creativity, wisdom, dignity, grandeur, devotion, peace, pride, mystery, independence, and magic.

You might have guessed it: it is the next generation of IBM Notes! Not one of the fixfeature packs, but an entire new generation. Under the influence Jason admitted: "We took the Notes Client source code and compiled it with XCode for iOS. Guess what: it worked. Cocoa needs some work, but it isn't rocket science"

There you have it! After banning Notes clients from your desktop, instead of relying on the IBM Client Application Access, you just can launch IBM Notes Native for iOS™ on your iPad and continue working. Any improvements and updates are automatically rolled out using the Apple app shop. Availability will be April 1st, 2018.

We live in interesting times!

01/04/2017

@Formula on JSON

Category

When you look at "modern" programming styles, you will find novel concepts like isomorphic (runs on client or server), Idempotency (same call, same result), Immutable (functions never mess with the parameters or global state) or map operations (working on a set of data without looping).

I put "modern" deliberately in quotes, since these ideas have been around since Lisp (or for the younger of you: since you sorted all blocks by colour and size in kindergarden). In the Lotus world we got our share of this with the venerable @Formula language (the functions, not the commands), IBM Notes inherited from Lotus 1-2-3. While it has served us well, so far it has been confined to the realm of the NSF.

Not any more! Thanks to Connections Pink and the ever ingenious Maureen Leland, @Formula will come to a JSON structure near you soon. As far as I understood the plan: each @Function will serve as an endpoint to a (serverless) microservice that executes on values provided, returning a new value object that can be chained to the next call stream style. I'm very excited about this new development. Watch out for news about Connections Livegrid™.

Time for Maureen to undust her blog!

11/01/2017

Serving Single Page Applications with Domino

Category  
Single Page Applications (SPA) are all the rage. They get developed with AngularJS, ReactJS or {insert-your-framework-of-choice}. Those share a few communialities:
  • the application is served by a static web server
  • data is provided via an API, typically reading/writing JSON via REST or graph
  • authentication is often long lasting (remember me...) based on JWT
  • authentication is highly flexible: login with {facebook|google|linkedin|twitter} or a corporate account. Increasingly 2 factor authentication is used (especially in Europe)
How does Domino fit into the picture with its integrated http stack, authentication and database? The answer isn't very straight forward. Bundling components creates ease of administration, but carries the risk that new technologies are implemented late (or not at all). For anything internet facing that's quite some risk. So here is what I would do:
Red/Green Zone layout for Domino

03/01/2017

Lessons from Project OrangeBox

Category   

Project OrangeBox, the Solr free search component, was launched after the experiments with Java8, Vert.x and RxJava in TPTSNBN concluded. With a certain promise we were working on a tight dead line and burned way more midnight oil than I would have wished for.

Anyway, I had the opportunity to work with great engineers and we shipped as promised. There are quite some lesson to be learned, here we go (in no specific order):

  • Co-locate
    The Verse team is spread over the globe: USA, Ireland, Belarus, China, Singapore and The Philippines. While this allows for 24x7 development, it also poses a substantial communications overhead. We made the largest jumps in both features and quality during and after co-location periods. So any sizable project needs to start and be interluded with co-location time. Team velocity will greatly benefit
  • No holy cows
    For VoP we slaughtered the "Verse is Solr" cow. That saved the Domino installed base a lot of investments in time and resources. Each project has its "holy cows": Interfaces, tool sets, "invaluable, immutable code", development pattern, processes. You have to be ready to challenge them by keeping a razor sharp focus on customer success. Watch out for Prima donnas (see next item)
  • No Prima Donnas
    As software engineers we are very prone to perceive our view of the world as the (only) correct one. After all we create some of it. In a team setting that's deadly. Self reflection and empathy are as critical to the success as technical skills and perseverance.
    Robert Sutton, one of my favourite Harvard authors, expresses that a little bolder.
    In short: A team can only be bigger than the sum of its members, when the individuals see themselves as members and are not hovering above it
  • Unit test are overrated
    I hear howling, read on. Like "A journey of a thousand miles begins with a single step" you can say: "Great software starts with a Unit Test". Begins, not: "Great software consists of Unit Tests". A great journey that only has steps ends tragically in death by starvation, thirst or evil events.
    Same applies to your test regime: You start with Unit tests, write code, pass it on to the next level of tests (module, integration, UI) etc. So unit tests are a "conditio sine qua non" in your test regime, but in no way sufficient
  • Test pyramid and good test data
    Starting with unit tests (we used JUnit and EasyMock), you move up to module tests. There, still written in JUnit, you check the correctness of higher combinations. Then you have API test for your REST API. Here we used Postman and its node.js integration Newman.
    Finally you need to test end-to-end including the UI. For that Selenium rules supreme. Why not e.g. PhantomJS? Selenium drives real browsers, so you can (automate) test against all rendering engines, which, as a fact of the matter, behave unsurprisingly different.
    One super critical insight: You need a good set of diverse test data, both expected and unexpected inputs in conjunction with the expected outputs. A good set of fringe data makes sure you catch challenges and border conditions early.
    Last not least: Have performance tests from the very beginning. We used both Rational Performance Tester (RPT) and Apache JMeter. RPT gives you a head start in creating tests, while JMeter's XML file based test cases were easier to share and manipulate. When you are short of test infrastructure (quite often the client running tests is the limiting factor) you can offload JMeter tests to Blazemeter or Flood.io
  • Measure, measure, measure
    You need to know where your code is spending its time in. We employed a number of tools to get good metrics. You want to look at averages, min, max and standard deviations of your calls. David even wrote a specific plugin to see the native calls (note open, design note open) or Java code would produce (This will result in future Java API improvements). The two main tools (besides watching the network tab in the browser) were New Relic with deep instrumentation into our Domino server's JVM and JAMon collecting live statistics (which you can query on the Domino console using show stats vop. Making measurements a default practise during code development makes your life much easier later on
  • No Work without ticket
    That might be the hardest part to implement. Any code item needs to be related to a ticket. For the search component we used Github Enterprise, pimped up with Zenhub.
    A very typical flow is: someone (analyst, scrum master, offering manager, project architect, etc.) "owns" the ticket system and tickets flow down. Sounds awfully like waterfall (and it is). Breaking free from this and turn to "the tickets are created by the developers and are the actual standup" greatly improves team velocity. This doesn't preclude creation of tickets by others, to fill a backlog or create and extend user stories. Look for the middle ground.
    We managed to get Github tickets to work with Eclipse which made it easy to create tickets on the fly. Once you are there you can visualize progress using Burn charts
  • Agile
    "Standup meeting every morning 9:30, no exception" - isn't agile. That's process strangling velocity. Spend some time to rediscover the heart of Agile and implement that.
    Typical traps to avoid:
    • use ticket (closings) as (sole) metric. It only discourages the us of the ticket system as ongoing documentation
    • insist on process over collaboration. A "standup meeting" could be just a Slack channel for most of the time. No need to spend time every day in a call or meeting, especially when the team is large
    • Code is final - it's not. Refactoring is part of the package - including refactoring the various tests
    • Isolate teams. If there isn't a lively exchange of progress, you end up with silo code. Requires mutual team respect
    • Track "percent complete". This lives on the fallacy of 100% being a fixed value. Track units of work left to do (and expect that to eventually rise during the project)
    • One way flow. If the people actually writing code can't participate in shaping user stories or create tickets, you have waterfall in disguise
    • Narrow user definitions and stories: I always cringe at the Scrum template for user stories: "As a ... I want ... because/in order to ...". There are two fallacies: first it presumes a linear, single actor flow, secondly it only describes what happens if it works. While it's a good start, adopting more complete use cases (the big brother of user stories) helps to keep the stories consistent. Go learn about Writing Effective Use Cases. The agile twist: A use case doesn't have to be complete to get started. Adjust and complete it as it evolves. Another little trap: The "users" in the user stories need to include: infrastructure managers, db admins, code maintainer, software testers etc. Basically anybody touching the app, not just final (business) users
    • No code reviews: looking at each other's code increases coherence in code style and accellerates bug squashing. Don't fall for the trap: productivity drops by 50% if 2 people stare at one screen - just the opposite happens
  • Big screens
    While co-located we squatted in booked conference rooms with whiteboard, postit walls and projectors. Some of the most efficient working hours were two or three pairs of eyes walking through code, both in source and debug mode. During quiet time (developers need ample of that. The Bose solution isn't enough), 27" or more inches of screen real estate boost productivity. At my home office I run a dual screen setup with more than one machine running (However, I have to admit: some of the code was written perched into a cattle class seat travelling between Singapore and the US)
  • Automate
    We used both Jenkins and Travis as our automation platform. The project used Maven to keep the project together. While Maven is a harsh mistress spending time to provide all automation targets proved invaluable.
    You have to configure your test regime carefully. Unit test should not only run on the CI environment, but on a developers workstation - for the code (s)he touches. A full integration test for VoP on the other hand, runs for a couple of hours. That's the task better left to the CI environment. Our Maven tasks included generating the (internal) website and the JavaDoc.
    Lesson learned: setting up a full CI environment is quite a task. Getting the repeatable datasets in place (especially when you have time sensitive tests like "provide emails from the last hour") can be tricky. Lesson 2: you will need more compute than expected, plan for parallel testing
  • Ownership
    David owned performance, Michael the build process, Raj the Query Parser, Christopher the test automation and myself the query strategy and core classes. It didn't mean: being the (l)only coder, but feeling responsible and taking the lead in the specific module. With the sense of ownership at the code level, we experienced a number of refactoring exercises, to the benefit of the result, that would never have happened if we followed Code Monkey style an analyst's or architect's blueprint.
As usual YMMV

30/12/2016

The totally inofficial guide to Verse on Premises

Category

Now that CNGD8ML is upon us, it is story time. Read about the why, who, what and what to watch out for.

To successfully deploy Verse, make sure to carefully read and implement the installation instructions. The availability of Verse makes Domino the most versatile eMail platform around, offering you the choice of: Notes Client, Outlook, POP2, IMAP4, iNotes, Verse, iOS, Android. Anywhay, here we go:

The back story

Verse on premises was a long (out)standing promise to the IBM customer base. Not everybody is ready to embrace the cloud, but interested in the new way to work. In SmartCloud Notes, the backend for Verse in the Cloud, all search is powered by Apache SOLR. If Verse got delivered as is, that would have required substantial hardware and skill investments for the on-premises customers.

So I made a bet with Michael Alexander, whom I worked with on TPTSNBN, that we could use standard Domino capabilities, not requiring Solr. Based on prototypes with vert.x and Java8 we gained confidence and got the go ahead to build the search component as OSGi plug-in (in Java6). So the search part (not the UI or other functionality) is on me.

The team(s)

There were two distinct teams working on the delivery of Verse on Premises (VoP): The core Verse team, that owns UI, functionality and features for both cloud and on premises and the search plugin team responsible to replace the Solr capabilities with native Domino calls.
The former is rather large, distributed between the US, Ireland and China. The later was led by the distinguished engineer David Byrd and just a few core coding members: David, Michael, Christopher, Raj and myself.
We were supported by a team of testers in Belarus and the Philippines. The test teams wrote hundreds of JUnit and Postman tests, just for the search API.

The Orangebox

Each project needs a good code name. The original Verse code name was Sequoia, which is reflected in the name of the plugins for core and UI functionality.

The search component, not being part of RealVerse™, needed a different name. In an initial high level diagram, outlining the architecture for management, the search component was drawn as an orange box. Since we "just" had to code "the orange box". The name stuck and led to our code name "Project OrangeBox" (PoB).
The inofficial Project Orange Box Logo
You can find Orangebox and POB in multiple places (including notes.ini variables and https calls the browser makes). So now you know where it is coming from.

13/09/2016

Domino meets RXJava

Category
Verse on premises (VoP) is nearing its second beta release and fellow Notes experts are wondering if they need to install Apache Solr as part of the VoP deployment. There was a lengthy, high quality discussion and quite some effort evaluating alternatives. In conclusion it was decided to deliver the subset of Solr capabilities needed for VoP as series of OSGi plugins to the Domino server. The project was formed out of the experience with ProjectCastle, which continues as Project OrangeBox to deliver these plugins. In VoP you might encounter one or the other reference to PoB, so now you know where it comes from.
One of the design challenges to solve was to emulate the facet results of the Solr search engine. I build some prototypes and finally settled on the use of RxJava.
RxJava is a member of the ReactiveX programming family, which is designed around the Observer pattern, iterators and functional programming. Check out the main site to get into the groove.
The task at hand is to convert something Domino (a ViewNavigator, a DocumentCollection or a Document) into something that emits subscribable events. I started with turning a document into an NotesItem emitter. Purpose of this was the creation of lighweight Java objects that contain the items I'm interested in. Since Domino's Java has special needs and I couldn't use the ODA, special precaution was needed.
There are plenty of methods to create an Observable and on first look Create looks most promising, but it left the question of recycling open. Luckily there is the Using method that creates a companion object that lives along the Observable and gets explicitly called when the Observable is done. To create the NotesItem emitting Observable I settled on the From method with an Iterable as source. The moving parts I had to create were class DocumentSource implements Iterable<Item> and class ItemIterator implements Iterator<Item>
Why Reactive? In a nutshell: a source emits data and any number of subscribers can subscribe to. Between the emission and subscription any number of filters, modifiers and aggregators can manipulate the data emitted. Since each of them lives in its own little class, testing and composition become very easy. Let's look at an example:
docSource.getItemStream(session).filter(nameFilter).map(toPobItem).map(nameMapper).subscribe(new ItemAdder());
You almost can read this aloud: "The source emits a stream of items, they get filtered by Name, then converted into another Java object (PobItem) and renamed before added to the subscriber.". In a different case you might want to collect all entities (users, groups, roles) that have access to a document, you migh create a "readerAuthorFilter". The individual classes are very easy to test. E.g. the Name filter looks like this:
// requiredFields is is a Collection<String> of NotesItem names to include or exclude
ItemNameFilter nameFilter = new ItemNameFilter(requiredFields, ItemNameFilter.FilterMode.INCLUDE);

public class ItemNameFilter implements Func1<Item, Boolean> {

    public enum FilterMode {
        INCLUDE, EXCLUDE;
    }

    private final Logger      logger      = Logger.getLogger(this.getClass().getName());
    private final Set<String> itemNameSet = new HashSet<String>();
    private final FilterMode  filterMode;

    /**
     * Flexible include or exclude
     * 
     * @param itemNames
     *            Collection of Names to include or exclude
     * @param filterMode
     *            INCLUDE or EXCLUDE
     */
    public ItemNameFilter(Collection<String> itemNames, FilterMode filterMode) {
        this.filterMode = filterMode;
        this.updateItemNames(itemNames);
    }

    public ItemNameFilter(Collection<String> itemNames) {
        this.filterMode = FilterMode.INCLUDE;
        this.updateItemNames(itemNames);
    }

    private void updateItemNames(Collection<String> itemNames) {
        this.itemNameSet.addAll(itemNames);
    }

    @Override
    public Boolean call(Item incomingItem) {
        // Include unless proven otherwise
        boolean result = true;
        try {
            String itemName = incomingItem.getName();
            boolean inList = this.itemNameSet.contains(itemName.toLowerCase());
            result = (inList && this.filterMode.equals(FilterMode.INCLUDE));
        } catch (NotesException e) {
            this.logger.log(Level.SEVERE, e);
            result = false;
        }
        return result;
    }
}

10/05/2016

Mach Dich auf die Socken!

Category  
A common requirement in corporate systems is "let me know when something is going on". In Notes we use "On document creation or update" triggered agents to process such events. To let external systems know about such a change R8 introduced the web service client. This works well in distributed system, but requires quite some work on both ends. In a recent case I had to optimize the communication between Domino and a task running on the same machine. The existing solution was polling the Domino API in short intervals for updates. Something I would call donkey mode. Sockets to the rescue. A few lines of Java in a triggered agent puts an end to donkey mode and provides the receiving end with all it needs in time:
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.Socket;

import lotus.domino.AgentBase;
import lotus.domino.AgentContext;
import lotus.domino.Database;
import lotus.domino.Document;
import lotus.domino.DocumentCollection;
import lotus.domino.NotesException;
import lotus.domino.Session;

import com.issc.castle.domino.Utils;

public class JavaAgent extends AgentBase {

	public static String	sockethost	= "127.0.0.1";
	public static int		socketport	= 1234;

	public void NotesMain() {
		Session session = null;
		AgentContext agentContext = null;
		Database db = null;
		DocumentCollection dc = null;
		Document doc = null;

		// The socket elements
		DataOutputStream out = null;
		BufferedReader in = null;
		Socket socketClient = null;
		try {
			// Get the Notes parts
			session = getSession();
			agentContext = session.getAgentContext();
			db = agentContext.getCurrentDatabase();
			dc = agentContext.getUnprocessedDocuments();

			// Get the socket
			socketClient = new Socket(sockethost, socketport);
			in = new BufferedReader(new InputStreamReader(socketClient.getInputStream()));
			out = new DataOutputStream(socketClient.getOutputStream());

			doc = dc.getFirstDocument();
			while (doc != null) {
				Document nextDoc = dc.getNextDocument(doc);
				this.signalOneDocument(doc, in, out);
				Utils.shred(doc);
				doc = nextDoc;
			}

			// Mark them done
			dc.updateAll();
		} catch (Exception e) {
			e.printStackTrace();
		} finally {
			Utils.shred(doc, dc, db, agentContext, session);
			// Close them
			try {
				if (out != null) {
					out.close();
				}
				if (in != null) {
					in.close();
				}
				if (socketClient != null) {
					socketClient.close();
				}
			} catch (IOException e) {
				e.printStackTrace();
			}
		}
	}

	private void signalOneDocument(final Document doc, final BufferedReader in, final DataOutputStream out) {
		try {
			String notesURL = doc.getNotesURL();
			out.writeBytes(notesURL);
			out.writeBytes("|");
		} catch (NotesException e) {
			e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		}

	}
}

No libraries to load, the only utility function used is Utils.shred() which is a error wrapped recycle call.
As usual YMMV
(Bad German pun in the title)

25/02/2016

Now we are token - Authorization using JSON Web Token in Domino

Category  
After having Vert.x and Domino co-exist, the door opens for a few interesting applications of the new found capabilites. One sticky point in each application landscape is authentication and authorization. This installment is about authorization.
The typical flow:
  1. you access a web resource
  2. provide some identity mechanism (in the simplest case: username and password)
  3. in exchange get some prove of identity
  4. that allows you to access protected resources.
In Basic authentication you have to provide that prove every time in form of an encoded username/password header. Since that limits you to username and password, all other mechanism provide you in return for your valid credentials with a "ticket" (technically a "Bearer Token") that opens access.
I tend to compare this with a movie theater: if you want to enter the room where the movie plays you need a ticket. The guy checking it, only is interested: is it valid for that show. He doesn't care if you paid in cash, with a card, got it as a present or won in a lucky draw. Did you buy it just now, or online or yesterday, he doesn't care. He cares only: is it valid. Same applies to our web servers.
In the IBM world the standard here is an LTPA token that gets delivered as cookie. Now cookies (besides being fattening) come with their own little set of trouble and are kind of frowned upon in contemporary web application development.
The current web API token darling is JSON Web Token (JWT). They are an interesting concept since they sign the data provided. Be clear: they don't encrypt it, so you need to be careful if you want to store sensitive information (and encrypt that first).

Now how to put that into Domino?

The sequence matches the typical flow:
  1. User authenticates with credentials
  2. server creates a JWT
  3. stores JWT and credentials in a map, so when the user comes back with the token, the original credentials can be retrieved
  4. delivers JWT to caller
  5. Caller uses JWT for next calls in the header
It isn't rocket science to get that to work.

18/02/2016

Vert.x and Domino

Category  
A while ago I shared how to use vert.x with a Notes client, which ultimately let me put an Angular face on my inbox and inspired the CrossWorlds project.
I revisited vert.x which is now 3.2.1 and no longer beta. On a Domino Linux server (I don't have Windows) and on a Mac Notes client the JVM is 64 Bit, which makes the configuration easier (no -w32 switch, no download of an additional JVM). The obligatory HelloWorld verticle ran quite nicely with my manually. However it wouldn't run, when the Domino ran using a startup script.
The simple reason: to be able to access the Domino instance the vert.x verticle needs to run with the same user as the Domino server. su into the user doesn't do the trick - and of course you can't login into my server with the id that runs Domino. The solution was to turn to the expert and his outstanding Linux boot script. Using the /etc/sysconfig/rc_domino_config_* file you can simply define the behavior of your Domino startup and shutdown experience. Mine looks like this (I use "domino" as my standard user, not "notes"):

rc_domino_config_domino

LOTUS=/opt/ibm/domino
DOMINO_DATA_PATH=/home/domino/notesdata
DOMINO_SHUTDOWN_TIMEOUT=600
DOMINO_CONFIGURED="yes"
BROADCAST_SHUTDOWN_MESSAGE="yes"
DOMINO_REMOVE_TEMPFILES="yes"
DOMINO_POST_STARTUP_SCRIPT=/home/domino/scripts/launch_vertx
DOMINO_PRE_SHUTDOWN_SCRIPT=/home/domino/scripts/stop_vertx

I have installed vert.x using npm using the full stack. With node.js installed, all you need is sudo npm install vertx3-full. Of course there are more conservative ways to install, vert.x, this may be an exercise left to the reader. I didn't use any of the environment variables exposed by the standard boot script to keep it independent. The script itself is just a few lines:

launch_vertx

#!/bin/sh
# Starts the vert.x tasks that talks to Domino
DOMINO_HOME=/opt/ibm/domino/notes/latest/linux
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export VERTX_HOME=/usr/lib/node_modules/vertx3-full/vertx
export DYLD_LIBRARY_PATH=$DOMINO_HOME
export LD_LIBRARY_PATH=$DOMINO_HOME
export CLASSPATH=.:$DOMINO_HOME/jvm/lib/ext/Notes.jar:$CLASSPATH
export NOTES_ENV=SERVER
vertx start com.ibm.issc.verseu.VerseLauncher -cp /home/domino/scripts/verseu.jar --vertx-id domino

The shutdown script is short and sweet. Since I used an vertx-id, I can use that to shut down the verticle without knowing or caring for its startclass name:

stop_vertx

#!/bin/sh
# Stops the vert.x tasks that talks to Domino
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export VERTX_HOME=/usr/lib/node_modules/vertx3-full/vertx
vertx stop domino

Next step: write some actual code beyond "Hello World".
As usual YMMV

13/02/2016

Disecting a mail UI

Category
At Connect 2016 Jeff announced that there will be an IBM Verse Client for Domino on premises. Domino customers are used to an high amount of flexibility, so the tempation will arise to customize the Verse experience.
However this ability is nothing that has been announced in any IBM roadmap. So any considerations are purely theoretical. What is the interface of Verse made of. WIthout reverse engineering, just by looking at it, one could come to the following conclusion:
A possible structure of the VerseUI
This looks quite managable and lean. Now if all these components are Widgets (if build in Dojo) or Directives (if build in AngularJS) or Components (an upcoming HTML standard) it wouldn't be too hard to envision customization abilities.
Using the excellent Tilt Extension for Firefox one can visualize how the different web clients are structured.

24/12/2015

Mail archive with Apache CouchDB / IBM Cloudant - Part 1

Category  
Like it or not, your eMail turned into the archive of your (working) past. One of the challenges with this archive is the tendency to switch eMail systems from time to time. IBM Notes won't open your Outlook PSD file, nor would Outlook open your Notes NSF database.
So a vendor and format neutral solution is required. The obvious choice here is MIME, which is for one, the format any message crossing the internet is encoded in, secondly all eMail applications support MIME - to some extend. Just storing each message into a directory structure isn't a good solution either, since navigation and search leave much to be desired, so some more work is needed.
Of course open standards tend to be ambiguous enough to allow different interpretation or the implementation of propriety extension. MIME is no exception. You can send any type of attachments, including malicious payloads, which are encoded and outside the MIME standard.
So looking at an archival solution here is my list of requirements:
  • Needs to be able to store MIME messages
  • Mime headers an other id fields need to be captured in database fields
  • Need to be able to sync on different locations for backup/availability
  • Need to be able to provide navigation access by sorted, filtered lists
  • Interface to do some analytics
  • Full text search
  • HTML and text content should be displayed directly, all other types should be listed as attachments
  • Inline images (href / src in the html content pointing to other mime parts) need to be dealt with
  • Import capabilities
  • Source code for inspection available, OpenSource if possible
Looking at the requirement, I concluded: I got a clear idea what I want to have, but I haven't found it. The logical next step: Let's build it.

04/11/2015

The lowdown on Notes application web and mobile enablement

Category
There are millions (if not billions) of lines of code written for the Notes client in small, large, simple, complex, epidermal and business critical applications. With the rise of browsers, tablets and mobile devices there is a need to web and mobile enable these applications, not at least since IBM won't let the C code of the Notes client run on iOS (which it could).
Whether we like it or not, that body of code can be considered technical debt.
  The strategy I see is mostly the same: classify applications by size, business criticality and perceived complexity. Then retire as much as possible and start a pilot for one (either insignificant or overly complex - mostly both) application using XPages. From there it goes nowhere.
Guess what. There's a better way.
Keep the part where you decide if an application can be retired. If you need to retain read access you could check out LDC Via, who remove the need to maintain a Domino infrastructure for those applications.
Then go and collect evidence (I know evidence based management isn't in fashion right now). There are elements in Notes that translate easily into web or mobile, parts that are hard and some stuff browsers can't do.
Collect how much you use of those, it makes decisions easier.

Design element and difficulty

  • View navigation:
    generally easy, its just data and Domino makes an excellent server for JSON data (I would render my own style, not the ?ReadViewEntries&Outputformat=JSON, but that's another story)
  • Categorised views:
    medium. The twisties we got so used to in Notes are not a native construct on the web (show me one that was web native). But with a little effort you can devise interaction patterns, go peek here.
  • View actions:
    HARD. They can be anything. Worst when code for front-end and back-end classes get mixed
  • Forms (read mode):
    Easy to medium. You could simply open a form using a browser and see something - or craft some XPage to show
  • Forms (edit mode):
    medium to hard (see below): You need to get your validation right and all the nice Hide-when conditions
  • RichText:
    HARD to impossible. RichText != HTML (and that's not even !==). For rendering you want to have Ben's enhancements as a start. RichText can contain attachments, wild formatting, OLE objects, Sections, Hidden stuff, tabbed or timed tables etc. All that doesn't really translate to HTML. To really know how RichText was used in your application and organisation you need to scan the actual data and analyse how RichText was used in each application
  • Code in forms:
    HARD. There is no LotusScript running in your browser, so you need to rewrite that logic. So better to know how many functions and lines of code you are looking at. Do your CoCoMo analysis. Code in fields (on enter/exit) needs to be re-though (nice word for retired)
  • Attachments:
    TEDIOUS. HTML doesn't know attachments, so you have to bridge between user expectations: can put an attachment anywhere in the text and the storage reality of HTML: simulate this by placing an image and a link. You might consider adding webDAV for a round trip editing experience
  • Reader & Author fields:
    Easy. Works
  • Encryption and Signature:
    (popular in approval applications) wait for Domino 9.0.2 - it supposedly will work there
  • Business Logic in hide-when formula:
    hard to find in Domino designer, DXL to the rescue
This list is probably not complete, but you get the gist

The approach

I would advocate (after the retirement of old applications):
  • Look at the whole application portfolio, not just one pilot. The best analogy: remember how long it took to drive your first 100m in a car (and I'm not talking about the 18 years wait to come of age). You had to take driving lessons (at least in civilized countries you have to), learn for the test, get a car (bargin with a parent?). So it took you around 20-40h, lets say: 20h. So you conclude to get to the supermarket, 3 km away it will take you 600h - so driving is out, you rather walk. Same applied to the "pilot". The first application is the hardest, so your experience wih it will screw the estimations.
    Furhermore looking at all applications gives you new synery potential and efficiency of scale
  • Create one module that handles view navigation for all applications (that probably would be a OSGi plug-in)
    When initially opening a document, it would point to a notes:// URL (you could have a little icon indicating that). Within a few weeks you have a working application giving some access to your Notes data (I would use JSON, AngularJS and Ionic for that app)
  • Next generate the forms using angular-formly or some {{mustache}} templates, so you have read-only access (DXL and XSLT skills come in handy)
  • Add a "request" functionality that lists the functions (Action buttons) an application once had, let users propose "I want this", so you get some priorities sorted out
  • Transit to contemporary development with a Domino backend
  • Do not fall into the trap: must look and work like a Notes client (we had that with the Java applets in R5 - last century). If someone is happy with the look of the Notes client - give them the IBM Client access plugin (PC only)

Why no magic button?

You could ask: why doesn't IBM provide this as an automatic module? After all they wrote LotusScript, the formula language and the Notes client. The short answer: IBM tried, we even had a prototype of LotusScript running on the JVM. The Notes client is such an incredible forgiving environment, any quality and mix of code "simply runs"™. I've seen code happily running (yes I'm talking about you 8000-lines-function) that was a melange of unspeakables.
Browsers on the other hand are fragile and unforgiving (not talking about quirks mode here). Any attempt to automate that never reached the stage of "point-and-shoot" - so it will not become an offering.

Why not just default and rewrite?

Good idea. Until you look closer. Domino's declarative security (ACL, Reader and Author fields anyone) is hard to replicate. The schema free Domino NoSQL store does neither map to Sharepoint nor an RDBMS (CouchDB could work), so you suddenly deal with data migration - a nightmare that can eat 80% of your total project cost. Lastly, the incremental approach outlined here allows to continue to use the applications (and eventually enhance them along the way), so disruption is minimised. Also the working Notes client provides a fallback for missing or malfunctioning features, this substantially lowers application risks.

Business partners to the rescue

You are not alone on this journey. IBM business partners and IBM Software Services for Collaboration are here to help. If you like the approach "I see Notes as one huge pool", you might want to talk to Redpill, they perfected this approach.

As usual YMMV.

27/07/2015

Adding Notes data to Websphere content manager via RSS

Category  
Websphere Content Manager (WCM) can aggregate information from various sources in many formats. A quick and dirty way to add Domino data is to use RSS. The simplest way is to add a page (not an XPage, a Page), define its content type as application/rss+xml and add a few lines to it:
<rss version="2.0">
	<channel>
		<title><Computed Value></title>
		<link><Computed Value>feed.xml</link>
		<description>Extraction of data from the Audience Governance database</description>
		<lastBuildDate><Computed Value></lastBuildDate>
		[Embedded view here]
	</channel>
</rss>

Thereafter create a view with passthrou HTML with all the values for an item element. Of course that is super boring, therefore you can use the following code to speed this up.

12/04/2015

email Dashboard for the rest of us - Part 2

Category  
In Part 1 I introduced a potential set of Java interfaces for the dashboard. In this installment I'll have a look on how to extract this data from a mail database. There are several considerations to be taken into account:
  • The source needs to supply data only from a defined range of dates - I will use 14 as an example
  • The type of entries needed are:
    • eMails
    • replies
    • Calendar entries
    • Followups I'm waiting for
    • Followups I need to action
  • Data needs to be in details and in summary (counts)
  • People involved come in Notes addresses, groups and internet addresses, they need to be dealt with
Since I have more than a hammer, I can split the data retrieval into different tooling. Dealing with names vs. groups is something best done with LDAP code or lookups into an address book. So I leave that to Java later on. Also running a counter when reading individual entries works quite well in Java.
Everything else, short of the icons for the people, can be supplied by a classis Notes view (your knowledge of formula language finally pays off).

11/04/2015

email Dashboard for the rest of us - Part 1

Category  
One of the cool new features of IBM Verse is the Collaboration Dashboard. Unfortunately not all of us can switch to Verse overnight, so I asked myself: can I have a dashboard in the trusted old Notes 9.0 client?
Building a dashboard requires 3 areas to be covered:
  1. What data to show
  2. Where does the data come from
  3. How should the data be visualised, including actionable options (that's the place we preferences between users will differ strongly)
For a collaboration dashboard I see 3 types of data: collaborators (who), summary data (e.g. number of unread eMails) and detail data (e.g. the next meeting). Eventually there could be a 4th type: collections of summary data (e.g. number of emails by category). In a first iteration I would like to see:
  • Number of unread eMails
  • Number of meetings left today
  • Number of waiting for actions
  • Number of action items
  • List of top collaborators
  • List of todays upcoming meetings
  • List of top waiting for actions
  • List of top action items
I'm sure there will be more numbers and list coming up when thinking about it, but that's a story for another time.

10/01/2015

Now that the client is gone - what to do with all that notes:// links?

Category
In your mind you drank the cool-aid and got onto IBM Verse or went to the dark side. You web enabled all of your applications (smartly) and you are ready to turn off the Notes clients.
But then you realize: there are notes:// URLs (or their file equivalents ndl files) and doclinks spread all over your organisation: in eMails, in office documents, in applications in personal information vaults.
So you wake up and go back to the drawing board.

What is the issue at hand?

Lotus Notes was one of the first applications (as I recall, the first commercial) that made large scale use of Hyperlinks (Yep, hyperlink doesn't mean it starts with http).
Creating, storing and exchanging links to information was one of the key success components of groupware and later the world wide web (which Notes predates). A hyperlink is unidirectional, so the link target has no knowledge where hyperlinks are located pointing to it. So inspecting a target gives no indication of a potential source.
When you move a target, e.g. change or decomission the server, then you create Link Rot, the digital equivalent of Alzheimer's disease.
Removing the Notes client adds a second problem. Each Hyperlink not only has a target, but also a protocol. That is the part in front of the colon.
Most of the time you think you deal only with http or https, but there are many more: mailto (for eMail), ftp (for file transfer), cifs (for file servers), ssh (for secure shell connections), call (for VoiP), chat (for chat), sap (to call the SAP client).
Each protocol has a handler application. When you don't have a handler application for a specific protocol installed, your operating system will throw and error when you try to use it.
Like a Demo? Try it for youself!
So to replace a protocol handler application (in our case the Notes client)

Solution

You need to do 2 things:
  1. In your web enablement / data migration project capture all source and target mappings. For some stuff simple formulas (regex) might do, but in most cases a key value store is required. Playing with different web technologies on wissel.net and notessensei.com, but rendering the same content, I tested the hypothesis and found key value stores suitable.
    In a nutshell: links need to live forever, come sing along
  2. Create a replacement for the notes:// URL handler (and the ndl file type), so any link starting with notes:// will then be looked up in your key value store and rendered by the appropriate application.
    The little special challenge here: when you web enable applications over a period of time, you still might have a Notes client, but that specific URL is gone. When you keep your NSF and just add the webUI, this won't be an issue. But it is a problem you will face when switching the platform

Denial

Alternatively you can claim ignorance is bliss and succumb to digital dementia. The usual aguments not to act:
  • the information is old, nobody needs it
  • we have great search in the new application, so users will find it
  • it is a waste of money and time
  • We have it archived, so what's the problem
  • We keep one machine with a client around, so users can look it up
If that's the prevalent line of thought in the organisation, wecome to binary lobodomy, if not, read on.

13/12/2014

Command line mail merge

Category
As shown before mail merging with Notes and {{mustache}} is simple and easy to code. To be able to run merging on the command line, I modified the bean and removed the XPages dependency and added a static main function. Now you can run it off the command line easily. It requires 5 parameters:
  1. the database name
  2. the CSV file
  3. a file with a HTML message including {{mustache}} tags
  4. a file with a Text message including {{mustache}} tags
  5. The senders eMail

01/12/2014

Poking around the iNotes HTTP API (Part 3) - retrieve message meta data

Category
So far we learned about the URLs, how to retrieve the Outline that contains folders and views and how to utilize JavaScript to deal with an API tha hadn't been designed for language neutral access. This part will deal with the retrieval of messages. There are quite some formats, strategies and tweaks to consider:
  • A message consists of headers (fields), and message parts that can be plain text, HTML, embedded images, attachments or other stuff (like the JSON used for embedded experiences). When to retrieve what part is an excellent question
  • Depending on your use case, your final client might or might not have access to the original iNotes implementation (Firewall, Offline, VPN), so just showing the HTML and let it fetch missing pieces might not do the trick
  • The HTML retrieved might not be well formed or contain links you don't want to execute (tracker images, tracker CSS, JavaScript stuff)
  • Automatically retrieving the whole message can be network heavy if there are large images or attachments in the MIME parts, this needs to be planned carefully
  • iNotes allows to retrieve the message fields as (kind-of) JSON using l_JSVars or retrieve the MIME headers using l_MailMessageHeader
  • The content of the message is accessible via s_MailMemoReadBodyContent or UNID/Body?OpenField or l_MailMessageHeader&PresetFields=FullMessage;1 (as raw Mime)
  • The MIME parts retrieved come infested with <br /> tags, since iNotes renders them in a display window (I would have used <pre>, but that's a different story. So they need cleanup
To begin I'll use the result of l_JSVars to learn about the document. Since, again, it is JavaScript, not JSON, I'll take the Rhino approach to extract a Java object.
The parts I'm interested in is in the variable named DXX which contains item that is a representation of NotesItems. True to the nature of an item, the JSON is quite elaborate, which you can see on the example of the form item:
{ "@name": "Form",
  "textlist": {
    "text": [
        {"0": "Memo"}
       ]
   }
}

For our purposes this will be flattened into a Map that has the item name as key. The Java code is quite similar to the retrieval of the Outline, the magic is in the JavaScript:
public MessageObject(String code, String rawMsg) {
    long start = System.currentTimeMillis();
    ScriptEngine engine = new ScriptEngineManager().getEngineByExtension("js");
    engine.eval(code);
    Invocable jsFunc = (Invocable) engine;
    MessageObject msg = (MessageObject) jsFunc.invokeFunction("getMessageObject", rawMsg);
    long end = System.currentTimeMillis();
    System.out.println(msg.getUNID());
    System.out.print("Time in ms:");
    System.out.println(end-start);
    return msg;
}

A little detail to stumble over is the item structure. It can be textlist/text or only text (feels like MIME Headers) and datetimelist/datetimepair/datetime or datetimelist/datetime.

27/11/2014

Poking around the iNotes HTTP API (Part 2) - Fun with Rhino

Category
The iNotes HTTP API wasn't designed for consumption outside the iNotes web client. This becomes apparent when looking at Form=l_GetOutline_JSON or Form=l_JSVars that return JavaScript and not JSON! The difference? JSON contains data only, while JavaScript contains function definitions too.
To make use of data included in the JavaScript there are 2 possibilities:
  • do some elaborate String operation
  • actually run JavaScript
Since Java 6, the JDK contains a JavaScript engine, that originally was developed as Mozilla Rhino. The most recent incarnation in Java 8 is Nashorn (which is German for Rhino). When you use an older JDK, think "older JS", nice functions like JSON.stringify or JSON.parse are missing and need to be added from elsewhere.
Turning the outline information into something Java useful we need to parse the returned JSON, lets see how that works. The Outline is a little more complex that it might seem. The JSON delivers 6 values (in an array) about the outline entries:
  1. The hierarchical sequence number. Something like 1 or 3.1 or 3.2.1 - depending on the level. The numbers have gaps where entries in the Outline are hidden from the web
  2. The hierarchy level starting with 0. So Asia\South-East\Singapore would be level 2
  3. The Label of the entry. Can be different from the actual name of the element
  4. The full path to the image for that entry. We'll strip that to get the name without a path, so we can use our own icons
  5. The full URL to that entry. It has the View Label (Name) and its UNID or system name in the string. We need the UNID for the unread count, so we'll extract that
  6. Can something be dragged there. 1 = Yes, it is a real folder, 0 = probably not, it is a view or a label only. "Probably" since e.g. you should be able to drag into the "Delete" view

24/11/2014

Poking around the iNotes HTTP API (Part 1)

Category
With the rise of web applications, something interesting happened: applications gained an observable API. Delivered over HTTP(s) with a few commands (GET, POST etc.) it became easier to find out "what is going on". Tools like Wireshark, Fiddler, Charlesproxy or TCPMon sit between your HTTP consuming client and the HTTP generating server. In browser applications the build in developer tools (Ctrl-Shift-I) provide the same level of access.
So anything flowing over HTTP is effectively an API. This led to the idea of the API economy (but that's a story for another time). Intercepting HTTP(s) traffic can be easier and more fun, than trying to customise customise a backend you might not have access to. (Of course there is evil interception too). Point is case, iNotes is quite a stack:
3d view of the iNotes client

So I poked around the iNotes API, this is what I found:

21/11/2014

Last usage of a mail file - for all users

Category
My admin is getting a little rusty. When I was asked: "How can I indentify a dormant mailbox?" I couldn't name a place in admin where to look. Of course, there is the NotesDatabase.LastModified property, but that would get updated on a design refresh too. So I asked innocently: "dormant, how?" After an amusing 2 hour discussion between the stake holders (only the popcorn was missing) we concluded that it is either
  • No document has been created for a while (including incoming)
  • No email has been sent for a while
Calendar entries or todo were not part of the consideration, but that's just a question in what view you look. For "any" document it is ($All) and for sent documents ($Sent). The LotusScript code is just a few lines. I needs to be run, so it can access all mailboxes.

20/11/2014

Providing user information in JSON

Category   
In the MUSE project we encountered the need to retrieve user information in JSON format. Easy done one would think. The trouble starts, when you have multiple directories and you need reasonable speed. Sometimes falling back to @Fomulas gives you what you need, fast and easy. @NameLookup knows where to look and you don't need any extra configuration. A simple call to an form will give you all you need: https://yourserver/somedb.nsf/namelookup?Readform for yourself or add &User=John Doe for any other user. This will return:
{
"QueryName": "John Doe",
"NotesName": "CN=John Doe/OU=ThePitt/O=GIJoe",
"AllNames": [
"CN=John Doe/OU=ThePitt/O=GIJoe",
"john.doe@thepitt.com",
"John Doe/ThePitt/GIJoe",
"John Doe",
"john.doe"
],
"eMail": "john.doe@thepitt.com",
"MailDomain": "SACMEA",
"MailServer": "CN=PittServer42/OU=ThePitt/O=GIJoe",
"MailFile": "mail/jdoe.NSF",
"Empnum": "0815",
"Empcc": "4711"
} 

The form makes extensive use of @NameLookup and looks in DXL quite simple.

30/09/2014

Put an angular face on your inbox

Category  
In the last instalment I got vert.x to emit a Notes view, read from your local mail file, to be emitted as JSON stream. While that might be perfectly fine for the inner geek, normal mortals want to look (and interact) with something more pretty.
The cool kids on the block for web interfaces and applications are Twitter BootStrap and AngularJS, so these will be the tools in this instalment.
Not familiar with them? Go and watch some videos. Back? Let's get going.
Since I'm not much of a designer, I choose the ready made Admin LTE template based on bootstrap. The main reason for choosing this one, was a timeline layout that I wanted for the inbox.
My inbox is already sorted by date, so it should fit nicely (so I thought). However the view optically is categorized by date and under the hood just a flat list of <li> elements. A little tweaking was necessary. The result looks quite OK for an after hour exercise:
Alternate layout for the Inbox
The steps to get there:
  1. Tweak the vert.x service to render a categorized by date version of the inbox. You can do that without touching the underlying view. I'll provide details on that modules in a later article (and some sample data below). The main difference for now: the inbox data will be available at the URL /notes/categorized/($Inbox)
  2. Create templates and directives for Angular
  3. Create the Angular.js app and its Controllers
Of course, no application is complete with a nice set of challenges. The biggest here was the flat list for the time line. I tried to adjust the CSS to accommodate a hierarchical list, where the outer elements are the date containing all messages arrived at that day, but there was too much CSS. So I decided to tweak the output a little.

21/09/2014

Creating nginx configurations for Domino SSL

Category  
Websites need to be secure, so the SHA-1 cipher is coming to an end. Despite best efforts, Domino is stuck with this outdated Cipher. While you can, on Windows, hide Domino behind IHS, I find nginx easier to tame.
Jesse explains how to configure nginx as the Domino proxy. So all is good, expecially since he also covered High availability.
But when you have a lot of sites, that's a lot of typing (and copy & paste from the Internet site documents). Mustache to the rescue! I've written about Mustache before and it suits the task quite nicely:
  1. Create one well working sample configuration
  2. Replace the site specific values with {{mustache}} variables
  3. Run it against all Internet site documents
The code I used (see below) generates just 4 variables:
  1. {{name}} The name of the site according to the configuration document. I use it here to configure the file name
  2. {{siteName}}The first web name, it will become the listen parameter
  3. {{allNames}} All web names, they will be listed as server_name
  4. {{settings}} all field values of the Internet site documents as concatenated strings. Using a dot notation they can be used directly. e.g. {{settings.SSLKeyFile}}. Using this approach you can do whatever is needed to generate your desired output
This is the initial template, based on Jesse's article:
server {
        listen {{siteName}}:443;
        server_name {{#allNames}} {{.}}{{/allNames}};
        client_max_body_size 100m;
 
        ssl on;
        # Original keyfile: {{settings.SSLKeyFile}}
        ssl_certificate      /etc/nginx/ssl/{{name}}.pem;
        ssl_certificate_key /etc/nginx/ssl/{{name}}.key;
 
        location / {
                proxy_read_timeout 240;
                proxy_pass http://localhost:8088;
                proxy_redirect off;
                proxy_buffering off;
 
                proxy_set_header        Host               $host;
                proxy_set_header        X-Forwarded-For    $proxy_add_x_forwarded_for;
                proxy_set_header        $WSRA              $remote_addr;
                proxy_set_header        $WSRH              $remote_addr;
                proxy_set_header        $WSSN              $host;
                proxy_set_header        $WSIS              True;
        }
}

The Java code takes in the file name of that template as parameter, so when you feel you rather use Apache or need a different output (e.g. a report), you are free to supply a different file here.

14/08/2014

Long Term Storage and Retention

Category  
Not just since Einstein time is relative. For a human brain anything above 3 seconds is long term. In IT this is a little more complex.

Once a work artefact is completed, it runs through a legal vetting and it either goes to medium or long term storage. I'll explain the difference in a second. This logical flow manifests itself in multiple ways in concrete implementations: Journaling (both eMail and databases), archival, backups, write-once copies. Quite often all artifacts go to medium term storage anyway and only make it into long term storage when the legal criteria are met. Criteria can be:
  • Corporate & Trade law (e.g. the typical period in Singapore is 5 years)
  • International law
  • Criminal law
  • Contractual obligations (E.g. in the airline industry all plane related artefacts need to be kept at least until the last of that plane family has retired. E.g. the Boing 747 family is in service for more than 40 years)
For a successful retention strategy three challenges need to be overcome:
  1. Data Extraction

    When your production system doesn't provide retention capabilities, how to get the data out? In Domino that's not an issue, since it does provide robust storage for 25 years (you still need to BACKUP data). However if you want a cross application solution, have a look at IBM's Content Collector family of products (Of course other vendor's have solutions too, but I'm not on their payroll)
  2. Findability

    Now an artifact is in the archive, how to find it? Both navigation and search need to be provided. Here a clever use of Meta data (who, what, when, where) makes the difference between a useful system and a Bit graveyard. Meta data isn't an abstract concept, but the ISO 16684-1:2012 standard. And - YES - it uses the Dublin core, not to confuse with Dublin's ale
  3. Consumability / Resillience

    Once you found an artifact, can you open and inspect it. This very much boils down to: do you have software that can read and render this file format?
The last item (and the second to some extend) make the difference between mid-term and long-term storage. In a mid-term storage system you presume that, short of potential version upgrades, your software landscape doesn't change and the original software is still actively available when a need for retrieval arises. Furthermore you expect your retention system to stay the same.
On the other hand, in a long-term storage scenario you can't rely on a specific software for either search or artifact rendering. So you need to plan a little more carefully. Most binary formats fall short of that challenge. Furthermore your artefacts must be able to "carry" their meta data, so a search application can rebuild an access index when needed. That is one of the reasons why airline maintenance manuals are stored in DITA rather than an office format (note: docx is not compliant to ISO/IEC 29500 Strict).
The problem domain is known as Digital Preservation and has a reference implementation and congressional attention.
In a nutshell: keep your data as XML, PDF/A or TIFF. MIME could work too, it is good with meta data after all and it is native to eMail. The MIME-Trap to avoid are MIME-parts that are propriety binary (e.g. your attached office document). So proceed with caution
Neither PST, OST nor NSF are long term storage suitable (you still can use the NSF as the search database)
To be fully sure a long term storage would retain the original format (if required) as well as a vendor independent format.

08/08/2014

Designing a REST API for eMail

Category  
Unencumbered by standards designed by committees I'm musing how a REST API would look like.
A REST API consists of 3 parts: the URI (~ URL for browser access), the verb and the payload. Since I'm looking at browser only access, the structured data payload format clearly will be JSON with the prose payload delivered in MIME format. I will worry about calendar and social data later on.
The verbs in REST are defined by the HTTP standard: , PUT, and DELETE. My base url would be http://localhost:8888/email and then continue with an additional part. Combined with the 4 horsemen verbs I envision the following action matrix:

06/08/2014

Running vert.x with the OpenNTF Domino API

Category  
In the first part I got vert.x 3.0 running with my local Notes client. The mastered challenges there were 32 Bit Java for the Notes client and the usual adjustment for the path variables. The adoption of the OpenNTF Domino API required a few steps more:
  1. Set 2 evironment variables:
    DYLD_LIBRARY_PATH=/opt/ibm/notes
    LD_LIBRARY_PATH=/opt/ibm/notes
  2. Add the following parameter to your Java command line:
    -Dnotes.binary=/opt/ibm/notes -Duser.dir=/home/stw/lotus/notes/data -Djava.library.path=/opt/ibm/notes
    Make sure that it is one line only. (Of course you will adjust the path to your environment, will you?)
  3. Add 4 JAR files to the classpath of your project runtime:
    • /opt/ibm/notes/jvm/lib/ext/Notes.jar
    • /opt/ibm/notes/framework/rcp/eclipse/plugins/
      com.ibm.icu.base_3.8.1.v20080530.jar
    • org.openntf.domino.jar
    • org.openntf.formula.jar
    I used the latest build of the later two jars from Nathan's branch, so make sure you have the latest. The ICU plug-in is based on the International Components for Unicode project and might get compiled into a future version of the Domino API.
Now the real fun begins. The classic Java API is conceptually single threaded with all Domino actions wrapped into NotesThread.sinitThread(); and NotesThread.stermThread(); to gain access to the Notes C API. For external applications (the ones running neither as XPage/OSGi nor as agent, the OpenNTF API provides the Domino Executor.

17/07/2014

Adventures with vert.x, 64Bit and the IBM Notes client

Category   
The rising star of web servers currently is node.js, not at least due to the cambrian explosion in available packages with a clever package management system and the fact that "Any application that can be written in JavaScript, will eventually be written in JavaScript" (according to Jeff Atwood).
When talking to IBM Domino or IBM Connections node.js allows for very elegant solutions using the REST APIs. However when talking to a IBM Notes client, it can't do much since an external program needs to be Java or COM, the later on Windows only.
I really like node.js event driven programming model, so I looked around. In result I found vert.x, which does to the JVM, what node.js does to Google's v8 JS runtime. Wikipedia decribes vert.x as "a polyglot event-driven application framework that runs on the Java Virtual Machine ". Vert.x is now an Eclipse project.
While node.js is tied to JavaScript, vert.x is polyglot and supports Java, JavaScript, CoffeeScript, Ruby, Python and Groovy with Scala and others under consideration.
Task one I tried to complete: run a verticle that simply displays the current local Notes user name. Of course exploring new stuff comes with its own set of surprises. As time of writing the stable version of vert.x is 2.1.1 with version 3.0 under heavy development.
Following the discussion, version 3.0 would introduce quite some changes in the API, so I decided to be brave and use the 3.0 development branch to explore.
The fun part: there is not much documentation for 3.x yet, while version 2.x is well covered in various books and the online documentation.
vert.x 3.x is at the edge of new and uses Lamda expressions, so just using Notes' Java 6 runtime was not an option. The Java 8 JRE was due to be installed. Luckily that is rather easy.
The class is rather simple, even after including Notes.jar, getting it to run (more below) not so much:
package com.notessensei.vertx.notes;

import io.vertx.core.Handler;
import io.vertx.core.Vertx;
import io.vertx.core.http.HttpServerOptions;
import io.vertx.core.http.HttpServerRequest;
import io.vertx.core.http.HttpServerResponse;

import java.io.IOException;

import lotus.domino.NotesFactory;
import lotus.domino.NotesThread;
import lotus.domino.Session;

public class Demo {
	public static void main(String[] args) throws IOException {
		new Demo();
		int quit = 0;
		while (quit != 113) { // Wait for a keypress
			System.out.println("Press q<Enter> to stop the verticle");
			quit = System.in.read();
		}
		System.out.println("Veticle terminated");
		System.exit(0);
	}

	private static final int listenport = 8111;

	public Demo() {
		Vertx vertx = Vertx.factory.createVertx();
		HttpServerOptions options = new HttpServerOptions();
		options.setPort(listenport);
		vertx.createHttpServer(options)
				.requestHandler(new Handler<HttpServerRequest>() {
					public void handle(HttpServerRequest req) {
						HttpServerResponse resp = req.response();
						resp.headers().set("Content-Type",
								"text/plain; charset=UTF-8");
						StringBuilder b = new StringBuilder();
						try {
							NotesThread.sinitThread();
							Session s = NotesFactory.createSession();
							b.append(s.getUserName());
							NotesThread.stermThread();
						} catch (Exception e) {
							e.printStackTrace();
							b.append(e.getMessage());
						}
						resp.writeStringAndEnd(b.toString());
					}
				}).listen();
	}
}

Starting the verticle looked promising, but once I pointed my browser to http://localhost:8111/ the fun began.

30/05/2014

Let's ditch IBM Notes and Domino

Category
Finally you decided it is time to move on, legacy no longer means "tried and tested" but "we need to move one" to you. After all you never really liked Cher.
Notes data is available via LotusScript, dotNet (through the COM bridge), in Java, Corba, C++, XML, REST, MIME, so how hard can it be? Actually not very hard, you just need to:
  1. Find sutiable replacement application platform(s)
  2. Rewrite the applications (don't dream: there is no such thing as "migrate an app")
  3. Migrate your users
  4. Migrate your data
... and then live happily ever after. Probably the easiest part is migrating the users, after all directories are more or less the same when you look at the LDAP side of them.
Migrating the users' encryption keys and secured documents could be considered "an application function", so you dodge a potential problem here.
Getting the data out, short of that pesky RichText content (yeah, the one with sections and embedded OLE controls), is easy - never mind reader and author access protection.
The transformation into the target format happens in a black box (the same the local magician uses for his tricks), you buy from a service provider, so that's covered, leaves platform and applications. (Bring your own <irony /> tags)
Lets have a look at the moving parts (Listing OpenSource components only, you can research commercial ones on your own):
What Domino does for you
  • Directory : OpenLDAP comes with most Linux distributions and is well understood. It doesn't take you down the propriarty extension hell (and you can remove obsolete schema parts) - it requires an RDBMS to run (not a blue one)
  • PKI : PrimeKey
  • Sync Server : For the database you could get away with the database, but you need one for mobile: Z-Push
  • Document Server : Pick any - or go with: Dropbox, Box, Copy etc.
  • Database Server : Notes is the mother of NoSQL, so you want to pick one of those: CouchDB, OrientDB or you go with DB/2 PureXML
  • Application Server : XPages is full fledged JSF, you you might opt for TomEE, but you will need some extra libraries. Or you ditch Java and adopt the cool kid on the block
  • Mail Server : the venerable SendMail does, as the name implies, send mail. Goes nicely with Mozilla Thunderbird
  • Web Server : That's easy. There's NGinX or Apache
  • Enterprise Integration : Might be quite tricky. Domino's DECS and LEI shuffle data from all sorts of sources. OpenESB might do the trick
  • Task scheduling : You can use OS level CRON (Just find a nice UI) or application level Quartz
Keep in mind, this replaces one server - Domino. Each of the tools needs to be:
  1. configured
  2. hardened
  3. populated with data
The ease of clustering you had with Domino, gets replaced with different methods and capabilities for each of the tools. Today Cloud is the new infrastructure, so you might get lucky with someone else configuring all of the above for you.

Once you got all the moving parts in place, you need to redevelop your apps. Don't be fooled, but run the analysis, to see the full magnitude of that task ahead.
As usual YMMV

17/04/2014

You want to move to Domino? You need a plan!

Category  
Cloud services are all en vogue, the hot kid on the block and irressitible. So you decided to move there, but you decided your luggage has to come along. And suddenly your realize, that flipping a switch won't do the trick. Now you need to listen to the expert.
The good folks at Amazon have compiled a table that gives you some idea how much it would take to transfer data:
Available Internet Connection Theoretical Min. Number of Days
to Transfer 1TB
at 80% Network Utilization
TB
T1 (1.544Mbps) 82 days
10Mbps 13 days
T3 (44.736Mbps) 3 days
100Mbps 1 to 2 days
1000Mbps Less than 1 day Some stuff for your math
(Reproduced without asking)
Talking to customers gung ho to move, I came across data volumes of 10-400 TB. Now go and check your pipe and do the math. A big bang, just flip the switch migration is out of the picture. You need a plan. Here is a cheat sheet to get you started:
  1. Create a database that contains all information of your existing users and how they will be once provisioned on Domino (If you are certified for IBM SmartCloud migrations IBM has one for you)
  2. Gather intelligence on data size and connection speed. Design your daily batch size accordingly
  3. Send a message to your existing users, where you collect their credentials securely and offer them a time slot for their migration. A good measure is to bundle your daily slots into bigger units of 3-7 days, so you have some wiggle room. Using some intelligent lookup you only present slots that have not been taken up
  4. Send a nice confirmation message with date and steps to be taken. Let the user know, that at cut-over day they can use the new mail system instantly, but it might take a while (replace "while" with "up to x hours" based on your measurements and the mail size intelligence you have gathered) before existing messages show up
  5. When the mailbox is due, send another message to let the user kick off the process (or confirm her consent that it kicks off). In that message it is a good idea to point to learning resources like the "what's new" summary or training videos or classes
  6. Once the migration is completed, send another message with some nice looking stats and thanking for your patience
  7. Communicate, Communicate, Communicate!
The checklist covers the user facing part of your migration. You still have to plan DNS cut-over, routing while moving, https access, redirection on mail links etc. Of course that list also applies for your pilot group/test run.
As usual: YMMV

17/04/2014

Domino Design Pattern: Secret documents

Category  
Domino's stronghold is security. However security is only as good as you design it. A frequent requirement in applications is to store a data set that is partially confidential and partially available for a wider audience. When you store these 2 data sets in one document, it isn't too hard to have the confidential information slip out:
  • using the document properties in a Notes client
  • using the document rest service
  • the property control from openNTF
In a nutshell: if you have 2 sets of data with different levels of read access requirements, don't store them in one document. A well working pattern in Domino is the "Secret Document". The following picture illustrates the concept:
Use 2 documents to store 2 sets of information security requirements
The user is presented with one form, but saving the entered data is done in two documents. The documents are cross referenced using the UNID. This can happen two way (as shown in the picture): the public document's UNID is saved in the secret document and vice versa - or - one way, with only the secret ID in the public document. A few pointers:
  • Based on the application's need some of the public data get repeated inside the secret document if that needs to be displayed on its own (e.g. a salary list in an HR application)
  • To avoid data drifting apart the respective data would only get updated in the public document ever and then copied to the secret document. In classic Notes that is done using a on-change agent, while in XPages a session-as-signer code snippet will suffice.
  • For very sensitive data (like even the normal user shall not see), these data sets could be stored in their own encrypted NSF. Then the UNID might not be enough, but the full notes:// url would make more sense
  • In classic Notes the embedded form editor makes the user experience with 2 documents seamless
  • In XPages two (or more) data sources sitting on one page will do the trick
As usual YMMV

17/04/2014

SmartCloud Notes little agent helper

Category
Now that we all drank the Cloud Computing CoolAid, we need to make it work. IBM's SmartCloud Notes looks enticing, since it offers 25G of eMail storage, way beyond what IT departments usually want to commit.
SmartCloud Notes even allows you customisation albeit within clear limits. So before you upload your extension forms you need to plan well.
One of the most unpleasant restrictions is: "No customer agents or scripts will be executed on server ", so no agent, no DOLS tasks. However you can run an agent (or other code) on an on-premises server. The interesting question is: when and how to trigger such code. Looking at the basic iNotes customization article you can find the Custom_Scene_PreSubmit_Lite JavaScript function. This could be the place to launch such a trigger. More on that in the next installment.
This article outlines the receiving end - the stuff that runs on your on-premises server. Instead of running agents, I'll outline a plug-in that allows to process the document submitted. The interface between SCN and this service is a JSON submission in the form of:
{
  "userName": "TestUser@acme.com",
  "action": "DemoTask",
  "unid": "32char-unid"
}

Once the plug-in receives this data, processing can commence. Of course the server (that's the ID the plug-in runs with) needs to have access to the mail file at the task required level. Let's get started:
In a plugin project we define a new servlet:
<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.4"?>
<plugin>
   <extension
         point="org.eclipse.equinox.http.registry.servlets">
      <servlet
            alias="/scntask"
            class="com.notessensei.cloudproxy.TaskServlet"
            load-on-startup="true">
      </servlet>
   </extension>
</plugin>

Then our servlet looks like this:
package com.notessensei.cloudproxy;

import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import lotus.domino.Base;
import lotus.domino.NotesException;

public class TaskServlet extends HttpServlet {

	private static final long		serialVersionUID	= 1L;

	// Number of threads allowed to run concurrently for data sync
	private static final int		THREADPOOLSIZE		= 16;

	// The background executor for talking to the cloud
	private final ExecutorService	service				= Executors.newFixedThreadPool(THREADPOOLSIZE);

	// The Cache where we keep our user lookup objects, handle with care!
	private final UserCache			userCache			= new UserCache();

	@Override
	protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
		// Takes in a JSON String and makes a task object
		InputStream in = req.getInputStream();
		CloudNotesTask it = CloudNotesTask.load(in);
		String result;

		if (it != null) {
			it.setUserCache(this.userCache);
			DocumentTaskFactory.getDocumentTask(it);
			this.service.execute(it);
			resp.setStatus(HttpServletResponse.SC_OK);
			result = "{\"status\" : \"task accepted\"}";
		} else {
			resp.setStatus(HttpServletResponse.SC_BAD_REQUEST);
			result = "{\"status\" : \"task failed\"}";
		}

		// Prepare the reply back
		resp.setContentType("application/json");
		PrintWriter out = resp.getWriter();
		out.println(result);
		out.close();

	}

	/**
	 * Get rid of all Notes objects
	 * 
	 * @param morituri
	 *            = the one designated to die, read your Caesar!
	 */
	public static void shred(final Base... morituri) {
		for (Base obsoleteObject : morituri) {
			if (obsoleteObject != null) {
				try {
					obsoleteObject.recycle();
				} catch (NotesException e) {
					// We don't care we want go get
					// rid of it anyway
				} finally {
					obsoleteObject = null;
				}
			}
		}
	}
}

26/02/2014

What is your investment in Notes applications - revisited

Category
About 2 years ago I asked: "What's your investment in Notes Applications?" and provided steps using DXLMagic to base the answer to that question on evidence rather opinion. With the arrival of version control capabilities in Domino Designer that task became easier (or different - your take). Revisiting the code base I devised new requirements:
  • The analysis should run against the On-Disk-Project rather than the NSF. The simplified assumption here: you deselected the "Use Binary DXL for source control operations"
    deselect binary DXL
    The "binary" format is stored in BASE64, so it wouldn't be impossible to decode, but quite some work to change the parsing, since the tags change too
  • The analysis code should have no Notes dependencies, so I can run on an Integration Server for continious measurements
  • It should be able to analyse a bunch of databases in one go
The result of this new requirements is cocomo.jar that allows you to simply run java -jar cocomo.jar DirectoryAboveYourOnDiskProjects ReportFile.csv.
Clarification (thx Stefan): DirectoryAboveYourOnDiskProjects is the location where all the directories of your individual On Disk projects are. So if you have C:\ODP\App1, C:\ODP\App2 and C:\ODP\OneMoreApp, you only run the tool once with java -jar cocomo.jar C:\ODP report.csv

The csv file contains one row per analysed database, with the last column being the Line of Code total for that application. That is the number you fill into the CoCoMo Tool in the New field. Then add Cost per Person-Month (Dollars) and hit Calculate. You get very enlightening results. After that, feel free to play with the settings of the tool.
On its first run the cocomo.jar writes out a properties file that defines what columns go into your csv report file, you can manipulate them as you deem fit. All the other settings are inside the jar, to peak into them, have a look at the source.
When your On Disk Projects are in different places, you can call the app with a 3rd parameter that points to a plain text file that simply lists directory names, one per line to be analysed: java -jar cocomo.jar DirectoryAboveYourOnDiskProjects ReportFile.csv FileWithDirectoryNamesOnePerLine

As usual: YMMV

13/02/2014

Domino Development - Back to Basics - Part 7: Map Reduce Domino Style

Category  
One of the odd things about Domino is the way things are called. It is Memo instead of eMail, Replication instead of Sync, Note store/Document store instead of NoSQL etc. The simple reason is the fact, that all these capabilities predate the more common terms and there was no label for them when Notes had them. In NoSQL circles MapReduce is a hot topic. Introduced by Google, now part of Apache Hadoop it can be found in MongoDB, Apache CouchDB and others. Interestingly it seems that short of Hadoop the mapping doesn't run distributed, but on a single server.
So what about Domino? Holding up the tradition of odd naming, the equivalent of MapReduce is Categorized View where View is the Map component and Categorize is the Reduce capability. The mapping part gets accomplished using the venerable @Formula (pronounced AT-formula) language which got inspired by LISP. If you ever wondered why you had to sort out all "yellow triangles" in kindergarden (a.k.a set theory), once you get started with @Formulas, that knowledge comes in handy. You mainly apply transformations using a set of @functions to lists of values (even a single value is a list, a list with one member). While there is a Loop construct, staying with sets/lists is highly efficient.
In its simplest case a column in a view is simply the name of the Notes item in a document (which will return an empty value if no such item exists). The formulas are used in 2 places: defining the column values of the data that is returned and selecting the documents to be included in the composition of the view. The default selection is SELECT @All which lists all documents in that database regardless of form used (if any) to create them or items contained in them. This is very different from tables/views in an RDBMS where each line item has the same content. Typically Notes application designers make sure, that all documents share a common set of items, so when combined in a view something useful can be shown. Often these item names are lifted from the mail template:Subject, Categories, From, Body.
Sorting resulting values is done as a property of the column (second tab in the property box), where sorting is from left to right. In classic Domino you can find views where specific columns are listed twice: once in the position where the user wants to see them, secondly in the sequence needed to sort, but with the attribute "hide this column". In XPages this is no longer necessary, since the view control allows to position columns in a different sequence than the underlying view.
When you categorize a column, typically starting at the first column, Domino offers a set of functions for the other columns to compute reduce values:
Reduce in Domino
You can access the values using ?OpenView&CollapseAll in Domino's REST API or in code using a ViewNavigator. A categorized view always starts with a category, so you begin with a .getFirst() followed by .GetNextSibling() for one level categories or .GetNextCategory() if you have multiple levels of categories. This capability helps when you aggregate data for graphics or pivot tables or [insert your idea here]

04/02/2014

Domino Development - Back to Basics - Part 6: Better safe than sorry - Security

Category  
Continuing from Part 5, this installment will shed a light on security.
Domino applications are protected by a hirarchical control system. If you fail to pass one hierachy level's test, it doesn't matter if a lower level would be compatible to you current credentials. E.g. when a database would allow anonymous access, but the server is configured to require authentication, you must authenticate. To fully understand the options, sit back and recall the difference between authentication and authorization. The former establishes who you are, the later what you can do. Let's look at both parts:

Authentication

When using a Notes client (including XPiNC) or running a Domino server, the identity is established using a public-private key challenge. The key is stored in the Notes.id file and (usually) protected by a password. So it qualifies as a 2 factor authentication (you need something, you know something). The beauty of this PKI approach is the availability of keys for signatures and encryption. Since users hate password entries you can perform unlocking of the Notes.id using various methods for single signon.
Accessing Domino through http(s) (or SMTP, IMAP and POP) uses open standards based authentication. I will focus on the http(s) options here. When a resource is protected (more on that later) the user is presented with an authentication challenge. In its simplest form it is http BASIC authentication that prompts for a username and password. In web UIs this isn't en vogue anymore, but in REST based access pulling and pushing JSON or XML it is still quite popular, since it is simple to use.
The most prevalent form (pun intended) is form based authentication. If a protected resource requires higher access than the current known user (who is anonymous before logging in), the user is redirected to a login page, where username and password are requested. This authentication can be per server or for a group of servers using LTPA or 3rd party plug-ins.
Users typically are listed in the Domino Directory or any compatible LDAP directory configured as authentication source. Users on Domino mail need to be in the Domino directory for mail routing to work.
A typical (structural) mistake: trying to use a remote 3rd party LDAP which creates additional network latency and single point of failure (the remote LDAP). When your Domino application server is running, your directory service is available, I wouldn't step away from this robustness. If your users strategically are maintained in a more fragile directory, use TDI to keep them in sync.
The final option to authenticate users is the use of X509 certificates. This requires the public X509 key to reside in the directory and the private key in the user's browser. I haven't seen that in large rollouts since it is a pain to administrate.
Anyway, as a result of an authentication, a user is identified with an X500 compatible name:
CN=Firstname LastName/OU=OrgUnit4/OU=OrgUnit3/OU=OrgUnit2/OU=OrgUnit1/O=Organisation/C=Country
Country (/C) and the Organisation Units (/OU) are optional. Typically we see 1-2 OrgUnits in use. They are practical to distinguish permissions and profiles as well as a remedy for duplicate names. If not taken care of carefully, users authenticated through a 3rd party LDAP will follow LDAP naming conventions, which is a PITA. When @UserName() doesn't start with CN= (unless it returns Anonymous) you need to take your admins to task.
In any case, it is transparent for an application developer on Domino how a user authenticates, you only need to care that it is happening when you protect a resource. Keep in mind: authentication is only secure via https!

Authorisation

Now you know who the user is, you define what (s)he can do. Domino uses 2 mechanism: the Access Control List (ACL) to define read/write access levels and the Execution Control List (ECL) to define what code a user can run. The first level of authorization is server access:

It doesn't matter what access a user might have inside a database, if the server won't let the user access its resources. So having your admin locking down the server properly is your first line of defense. Server access is defined in the server document that also contains the ECL for the server. A normal user doesn't need ECL permissions, only the ID that is associated with the XPage (or agent) that a user wants to run. "Associated" here means: the ID the code was signed with. By default every time a developer saves an XPage, that page gets signed with the developer's ID. This is very different from file based application servers where a file doesn't contain a signature, but similar to the JAR signing in the Java world. Keep in mind: in Java that is a manual process using a command line tool and requires the acquisition of an (expensive) code signing certificate, while in Domino it is automatic.
Common practise however is, not to allow a developer to run code on a production server (unless you are Paul), but signing the database with either the server.id or a specific signer ID. This step would be performed by your admin.
When you access an NSF, the access level is set in the ACL to one of these values: No Access, Depositor, Reader, Author, Editor, Designer, Manager.
Additionally a user can have roles assigned to her, that allow a refinement of interaction and access. Roles are always used with the name in square brackets. Typical roles are [server] or [Admin]. Roles are defined per NSF. The following tables shows the permitted operations per access level:

Read Access


Having read access to a database doesn't automatically make all documents visible. They still can be protected by reader fields. More below

Write Access


Having Author access to the database does enable a user to edit any document where the user is listed in one or more items of type author, While Editor access allows to edit any document the user can see. So as a rule of thumb:
The access level for normal users to a Notes/Domino based application should be set to Author

Reader & Author fields

They are quite unique to Domino, since they allow declarative protection of a note. If you implement (in code) something similar in an RDBMS, you get quite a complex relation:

Looking at the above table you could conclude: Author fields are only relevant for users with exactly Author access to a database: if the access is lower they won't be able to edit anything, if the access is higher, they can edit anything. However in conjunction with Reader fields they have a second purpose. Reader protection is defined as:
When a document has at least one item of type Readers that has content (non-empty), then access to that document is restricted to users and servers that are explicitly (by name) or implicitly (by group membership, role or wildcard) listed in any of the Reader or Author items.
A typical pattern in a Notes application would be a computed Author field with the value [server]. The application server would get the role in the ACL. Since a server most likely has Manager access anyhow, it has initially no impact on the application. However when some mechanism activates reader protection, this field ensures that the server can see all documents. It is important to use an Author field/item here, to avoid triggering read protection where it isn't necessary. Reader fields have a performance cost, so use them wisely.
A popular mechanism is to provide a checkbox "Confidential" and then compute a reader field: @if(Confidential!="";DocumentPeople;""), where "DocumentPeople" would be an item of type Names. In workflow applications, the content of reader and author fields isn't static, but changes with the progress of the workflow. Careful planning is required.

There is more

Notes can refine access further. Documents can be encrypted and signed, as well as sections in a document can be access controlled or signed. However those capabilities are not yet exposed to the XPages API and are thus of limited interest for new XPages developers.

Further readings

There are a number of related articles I wrote over the years: Questions? Leave a comment or ask on Stackoverflow

15/01/2014

GMail2Notes in less than 300 lines

Category
I admit, the headline is an attention grabber
The total number of lines is, of cause, longer. For one, I used previous code, as well e.printStackTrace() isn't error handling.
Moving data between two places is a favorite IT pasttime (believe me, there are worse ways to kill time). Since I made friends with MIME Messages before, I was wondering how that would fit together with gMail. Every eMail system has its little "specialties". To connect to gMail I used the IMAP with username and password route (which is the only one I know of for IMAP).
Most of the Googles point to "reading the inbox" which is a distinct different task from: "pulling out all your eMail". When connecting to gMail using the enhanced IMAP classes you can pull all "Folders" using store.getDefaultFolder().list(). There a particular folder is interesting: [Gmail]. In Notes speak I would say: it contains the system views, most notably "All documents", "Sent" and "Spam".
To capture all documents going after that one view is sufficient. Google abandoned the idea of "folders" and replaced them with "labels". The idea there was: a document can be in one folder only, but can have many different labels. For other email systems that might hold true, but the way folders a implemented in Notes nicely fits with the idea of labels.
Each message comes with a getLabels() function that directly maps (with a little attention) to a NotesDocument.putInFolder(label) call. The code isn't perfect, but can be a good starting point for your own adventure.

06/01/2014

Using a web service to send an eMail

Category
In an integration project with other platforms the need arose to send messages that are triggered by a web service. In Domino this is just a few lines of code. Added bonus: the message is fully "embedded experiences" enabled. This is the LotusScript class:
Public Class SendADocumentWithPDFAttachment
    
    Public sendTo As String
    Public subject As String
    Public textBody As String
    Public htmlBody As String
    Public callBackURL As String
    Public attachmentAsMime As String
    
    Public Function sendMessage(message As SendADocumentWithPDFAttachment) As String
        Dim s As New NotesSession
        Dim db As NotesDatabase
        Dim doc As NotesDocument        
        Dim header As NotesMIMEHeader
        Dim body As NotesMIMEEntity
        
        On Error Goto Err_sendMessage
        
        'We are dealing with a MINME message here!
        s.Convertmime = False
        Set db = s.Currentdatabase
        Set doc = db.Createdocument()
        Call doc.Replaceitemvalue("Form", "Memo")
        Call doc.Replaceitemvalue("From", s.Username)
        Call doc.Replaceitemvalue("Subject", message.subject)
        Call doc.Replaceitemvalue("SendTo", message.sendTo)
        'Add other fields as you deem fit
        
        'Prepare the message body
        Set body = doc.Createmimeentity("Body")
        body.Createheader("Content-Type").Setheaderval("multipart/alternative")
        body.Createheader("MIME-Version").Setheaderval("1.0")
        body.Createheader("Subject").Setheaderval(message.subject)
        
        'Adding the content
        If message.textBody <> "" Then
            Call Me.addContent(s, body, message.textBody,"text/plain;charset=UTF-8")
        End If  
        If message.htmlBody <> "" Then
            Call Me.addContent(s, body, message.htmlBody,"text/html;charset=UTF-8")
        End If
        If message.callBackURL <> "" Then
            Call Me.addContent(s, body,|{ "url"  : "|+message.callBackURL+|" }|,"application/embed+json;charset=UTF-8")
        End If
        Call doc.Save(True, True)
        
        'Sending has 2 options: use the send() function or copy the document
        'into the mail.box. We use send here
        Call doc.send(False)
        s.Convertmime = True
        
        sendMessage = "Message submitted"
Exit_sendMessage:
        Exit Function
        
Err_sendMessage:
        sendMessage = Error$
        Resume Exit_sendMessage
        
    End Function
    
    Private Sub addContent(session As NotesSession, body As NotesMIMEEntity, content As String, contentType As String)
        Dim mime As NotesMIMEEntity
        Dim contentStream As NotesStream
        On Error Goto Err_addContent
        Set contentStream = session.Createstream()
        Set mime = body.Createchildentity()
        Call contentStream.Writetext(content)
        contentStream.Position = 0
        Call mime.setcontentFromText(contentStream,contentType,ENC_BASE64)
Exit_addContent:
        Exit Sub
        
Err_addContent:
        Print Error$
        Resume Exit_addContent
    End Sub 
End Class

02/01/2014

Domino Development - Back to Basics - Part 5: Finding data - Collections and Search

Category  
Continuing from Part 4, this installment will clarify finding data. It, again, is different from other database concepts. Read on:
In a RDBMS your data retrieval queries are formulated in SQL, while XML, RDF or Graph databases might use sparQL, Gremlin, Cypher or JavaScript. There is no shortage of Query languages (one you should master in any case is XPath, get the full reference from the inventor himself).
On a first look a Query Language seems absent from IBM Notes. On a second look, there are two.

The IBM Notes Query Languages

Access to Notes data mostly happens using a view, so the pre-selection happens using the SELECT statement and the @Formula language. We will see how to narrow results down, just below. But you are not limited to creating a view first, a database can be queried directly using @Formula when executing Database.search("some @Formula expression"). This works like any other query language around. This flexibility comes with a price: speed. Database.search is the slowest way to retrieve data from a NSF and you shouldn't use that for your default access path (that is: the actions most users will use in a regular manner).
The second query language is the composition of a fulltext query. In its simplest form it is just some text you want to find, but it can be extended to specify the fields to look into and construct additional constraints. It takes a little to get used to it, but it is quite powerful, just one example: fish SENTENCE chips finds documents where fish & chips are mentioned in the same sentence. Important note here: make sure the FullText index is configured, so it isn't crawling slow.

Finding data

There are several ways data retrieval happens in IBM Notes:
  1. Direct access
    Notes documents can be addressed using a notes:// address. The session class has a resolve(notesurl) method (not the fastest) and the database class has Database.getDocumentByUNID(id). Don't confuse it with Database.getDocumentByURL(...) which created a document based on HTML content at the specified URL. Internally the byUNID is used whenever you click on a Notes document link. It is one of the fastest access methods. There is also byID, which you should avoid, since it isn't save across replicas or copy style compaction
  2. View/Folder access
    The view selection formula (for folders: action taked by users or code) defines a set of documents shown in a view. Using one of the collections (see below) you can process all those using View.getAllEntries, View.createViewNav() or by directly starting to loop through the documents using View.getFirstDocument() (or getFirstEntry()). When looping through them, any saving back of a document might remove it from the view index, so you need to fetch the next document/entry before that
  3. Subsets of View/Folder
    This is the most typical query access to Notes data. To make this work the view needs to be sorted or categorized (depending on the method). A typical example in a workflow application would be a view with the selection formula SELECT Status="Pending Approval" and the first column categorized with CurrentApprover. You would retrieve documents using View.getAllDocumentsByKey(...). This method works already with sorted columns (don't need to be categorized) and can take a Vector as search term. It would then match the Vector elements to the sorted columns for search.
    However using categorization has the advantage, that you can directly address that view in an URL using ?OpenView&RestrictToCategory=search term and the view control has a matching property as well. An alternate approach is to use View.getDocumentByKey(...) to jump to the first Document/Entry and then loop until the desired end. This approach is useful when you might not process all entries or process beyond the current key (e.g. process all requests from R-Z)
  4. FullText search
    Make sure you actually have a FTIndex on the database, otherwise that will be slow. The syntax takes a while to get used to. There is one FTIndex per database, even while you can limit the search to one view
  5. Database.Search(...)
    The most flexible search, with the price of speed and sort order: slow and you have to sort it yourself. Don't use unless unavoidable
Depending on your needs pick one or the other

Collections

Depending on your access route choosen above, you might get one of the 4 result collections. Those are no proper Java Collections, but come in their own little API implementation, unless you use a well designed Notes API. You will find getFirst and getNext(currentOne) methods
  • DocumentCollection
    The oldest and most flexible collection. Can originate from a database, a view, a search. Since you have full access to all documents in it, it is the slowest collection. It was also the first collection available in the API and is a good candidate for being replaced by a ViewEntryCollection when tuning your database. A DocumentCollection contains only data documents. Is has fancy set operations
  • NoteCollection
    The NoteCollection can contain data and design Note elements and is slightly faster to build than the DocumentCollection, since it initializes a Note class only on access. So the total time "creation + access" is similar to the DocumentCollection. The main use case is access to design elements and some fancy set operations
  • ViewEntryCollection
    This collection provides all (or selected) documents from a view. Instead of opening each document it gives access to the column values in the ViewEntry object, as well as the same fancy set operations you learned to love from the DocumentCollection and the NoteCollection. The ViewEntryCollection performs faster, given you have all values you need in a view column
  • ViewNavigator
    Technically it isn't a collection, but the ability to point around in the view index. So you won't get any set operation to refine the result, but you get speed. You can jump from category to category (great for reading the reduce() values) and run through an entire view very very quickly. The ViewNavigator is a main reason to plan your views well

Next up: Better save than sorry - Security

26/12/2013

Domino Development - Back to Basics - Part 4: Domino views are different

Category  
Continuing from Part 3, this part is typically the hardest to understand when coming from an RDBMS background. So take your time.

Domino Views are different

In Domino data is typically accessed via a view, but views are different than the ones you know in an RDBMS. The following table should provide an rough overview.
ItemRDBMSDomino
Data Defined in a database schema. Data is contained in tables. All records in a table are uniformly the same. Fields without values are there but empty. Different types of data require different tables. New data requires ALTER TABLE statements wich affects existing data (and often requires down time). Each column can have one value Data is contained in documents. Documents are schema free (there is a Meta Schema). Items in documents (the closest to a column in a table) can have multiple values. Each document can have a different set of items. New data is added as needed, no downtime or change of existing documents required
Views Selection of rows, often accompanied with a JOIN operation to denormalize normalized data. Can be all data of a table or a subset, picked by a WHERE part in the SELECT statement.
By definition views don't contain any data, but pull them ad hoc from the participating tables.
Various RDBMS systems use indexes on keys used in SQL statements to improve performance. The ad-hoc nature of the queries offers maximum flexibility at CPU and I/O cost.
There is a whole industry around SQL Query Optimization (and there is the whole story about SQL Injection attacks even by mothers)
  • are pre-created and updated by a view indexer task when documents are created or updated.
  • They do occupy space inside an NSF (there are discussions in development to externalize them to reduce backup load and allow more I/O tuning).
  • Since they don't require a document to be opened, they are fast (unless you use performance killers - see below)
  • Each column in a view shows data. Using Domino designer that data can be different for each row (defined by @Formula
  • Since items can have multiple values, they can appear more than once in a view (controlled by the view design)
  • Different document types (e.g. a customer entry and an order entry) can show up in the same view without the need for a JOIN operation (which anyway isn't a feature of a Notes view - this is one more reason why Notes favours inheritance over relation)
  • Accessing a view using a ViewNavigator is very fast
  • A view can contain data that is computed and not contained in the documents. Typical uses are translations of status codes to human readable text)
Hierarchy SQL Tables and views are flat by definition. Some SQL extensions (e.g. Oracle) allow hierarchical queries, but the result will be a flat query result.
  • views can be flat or hierarchical
  • A common use for hierachical views is the option to categorize a column value. This operation will sort the column and show each value once only above the result rows (in the UI typically with a twistie icon)
  • A categorized column also can perform some additional simple Reduce operations (albeit not distributed)
  • Programmatic categorized views are typically used with @DBLookup; @DBColumn; getAll[Documents|Entries]byKey
  • Multiple columns can be categorized
  • Typically the categorized columns are the first ones on the left, but that isn't necessary
  • Subcategories can be created by having the item value containing a backslash. If an item contains more than one value, the document will appear in multiple categories. An alternative approach turns multiple values into subcategories: @Implode(fieldname;"\\")
  • Besides categorization there is a secondary hierarchy mechanism available: the response hierarchy. Defined by a view property, response documents (of all levels) are shown automatically below their respective main document. A special column can be defined that is shown only for response type documents (containing a valid $REF item). In this case only the columns left of this are shared with main documents, columns right of it are ignored for responses and only the response column is shown (check the discussion template for how it looks)
Sorting Sorting happens using an ORDER BY clause. For resorting a new select clause needs to be issued. RDBMS systems allow indexes to be created to improve sorting speed.
  • Sorting is a property of a view column. It will increase the view index size and thus I/O and reindexing time
  • Sorting happens from left to right
  • Sorted columns don't need to be visible in the view (there is a property to hide columns conditionally or in any case)
  • Columns can be resorted in the UI when the respective property is set. The custom sorting sequence is also available as URL parameter
  • A common design mistake (creating index data unused): make every column sorted, even if the sorting has no more effect on the result sequence and/or make every column resortable even if users might not use it. For the later case there is a remedy: set a column property to defer the creation of the alternate index until it is requested the first time
Definition Views are defined in SQL using any editor of your choosing and then uploaded/executed in the database. Database admin rights are required Views are defined in Domino Designer or even the standard Notes client. Users can be allowed to create "private" views only visible to themselves, or at access level Editor views that can be shared. Columns are defined by picking item names from a list, predefined functions or writing @Formula a LISP like language (easy to learn, quite powerful to use). Brave souls have been spotted using DXL to write view definitions in XML and upload them into designer (but you don't want to do that)
Selection View entries are selected by WHERE in SQL
  • The selection statement also start with SELECT, but uses the @Formula language for picking the documents
  • All documents (typically in a view named ($All)) are selected by SELECT @All
  • More samples: all customer documents by SELECT Form="Customer", incomplete workflows by SELECT @IsMember(Status;"Submitted":"In Progress")
  • Overly complex select statements slow down view computation and you might need to employ some brute force view tuning
  • A performance tip is to compute the criterion for selection inside the document to result in a field value @True|@False and then limit the selection formula to SELECT showInView_Pending
Performance killers There is nothing better to make your hardware seller more happy than code that violates performance considerations. In SQL the typical performance killer are OUTER JOINS especially over many large tables. Missing indexes are another cause Since Notes views contain actual data, they need to be treated a little different. There are a few killers around:
  1. Performance killer number one is the use of time based view selection formula: @Now; @Today; @Yesterday; @Tomorrow. In one word DON'T. IBM isn't so strict, but I am. You do have options
  2. The second killer is the use of @IsResponseDoc as part of a view selection when you end up with only a few parent documents, show documents in response hierarchy and many "hidden" responses since the parent isn't displayed. Better use @AllDecendants, check the explanation for more details
  3. As mentioned above: be stingy with sorting of columns, both presorted and optional. Make sure if you define optional sorting, that you defer creation until first use
  4. Overly complex view selection formulas and too many views slow down the view indexing process, so take only as much as you really need. In classic Notes we often find countless variations of views for display. In XPages you can often combine them (given they differ only in columns) and use the XPage for the variation in look and feel
  5. Last not least: your code. Let the view indexer do its job. If you use view.refresh() everywhere "as a precaution" you pay dearly with sucking performance. A view gets locked when reindexed, so 100 users all executing a view.refresh() lock the view 100 times sequentially
Watch your SQL

Next up: Finding data - Collections and Search

25/12/2013

Domino Development - Back to Basics - Part 3: Not all Documents are created equally

Category  
Continuing from Part 2 this installment will introduce you to the four types of documents you can find inside an NSF:
  1. Documents
  2. Responses
  3. Response to Responses
  4. Profiles

Profile Documents

All document types follow the same rules as outlined in Part 2, but differ in the way they are created or accessed. Short of profiles all documents typically get accessed in one or more views, by users or your code. Profiles on the other hand, never show up in a view, they are only accessed using some code. The easiest case is @Commmand([EditProfile]"NameOfAForm") for a System profile and @Commmand([EditProfile]"NameOfAForm";@UserName) for a user profile. Notes keeps an internal index for profiles and they are typically cached in memory, so accessing them is superfast (and quite a headache if you fall into the trap to use them for frequently updating information). Their intended use is storing configuration and selection options, so @GetProfileField(FormName;FieldName) has rapid access to options. In XPages applications the need for profiles is diminished, since you typically would store such configuration options in the Session and Application Scope where you load them once.

Document hierarchies

The Document Hierarchy
The different "Document types" are actually "only" form properties (One can see how forms and documents are points of confusion). Inside a document the presence of the $REF field (containing the UNID of the parent) makes a document a response. If the parent document happens to be a response itself, the document then is considered a "Response to Response" All Notes APIs have methods to create or query a Parent - Child hierarchy. A good overview can be found in this Ytria TechLab. To my knowledge a Response-to-response document doesn't contain a direct reference to the root document
The form properties are used when you create a document in the Notes client to automatically establish Parent-Response relationships. A typical mechanism used in response document forms is to enable inheritance (that feature can also be used for standard documents, it isn't limited to responses!): When the document get created in the context of the parent (think: you read an entry in a discussion and press reply) all items from the parent document, that have fields in the response form with the same name as the item get copied into the response document (good fun if item and field don't have the same data type). Additionally all formulas computation in the response get executed against the parent, but stored in the response. This happens once at creation, not thereafter. To keep fields in sync you would use @GetDocField($REF;"SomeFieldName") to obtain the latest value (there are methods in LotusScript, JavaScript and Java too).
Sounds complicated? Go and create a discussion database (the template is on your Notes client) and play with it for a while. All this leads to an important Notes concept:

Inheritance over Normalization

Database normalization is a corner stone concept of RDBMS. If RDBMS is all you ever used and you are stuck in the idea that databases = RDBMS (really?), this will be hard to swallow.
Normalization is an abstraction that is suitable and needed for an RDBMS, it isn't a real life fact or an absolute necessity. If you blindly apply Normalization to XML Database, document databases, Graph databases you create more problems than solutions. So free your mind and follow:
Let's look at a typical normalization example: an invoice: You have one customer table, one invoice table, one items table and one products table. In your invoice table you point to the customer table, in the item table to your invoice table and your product table. Life seems good.
Invoice Entity relationship diagram
But then pesky reality kicks in: Customers move, but legal demands that existing invoices don't change (commonly changing a legal document is called forgery). Product prices change, or sales negotiated a different price. Customers might call your articles different from your product table etc. So you adjust your diagram:
Enhanced Invoice Entity relationship diagram
The attributes in red are inherited from the respective tables. This inheritance in an RDBMS is entirely done in code, in (classic) Notes it could be done through a form property. So inheritance is also quite common in RDBMS based applications too.
In Notes it is rather the rule than the exception. Retrofitting an RDBMS normalization onto a Notes application might not bring the benefits you might expect, so keep in mind:
Notes favours inheritance over normalization.
Interestingly when you want to export data, e.g. for data interchange you hardly generate one table/file per RDBMS table, but you de-normalize the data for transport. In Notes this tandem of normalization/de-normalization isn't necessary since you deal with a (self contained) document already.

Items have multi-values and can be grouped

A typical way in a Notes application to handle parent-child records is the use of a group of related multi-value fields (in the form), resulting in a set of items in the document with the same number of values. You might have one address field (which can have many lines - much more flexible than a text column in an RDBMS) and a set of multi-value fields like: ProductID, description, Price, Quantity and SumPrice. Since the (classic) UI didn't make it very easy to keep those in sync, a module called Table Walker had been made available on the Sandbox. I later created Tablewalker for XPages as part of the XPages tutorial (with enlightened bug fixes courtesy Matt White). Keeping the line items inside a document makes transporting, processing or securing them much easier, however it makes querying the line items a bit more difficult (more on that when looking at views). You can take a similar (actually more powerful from a modelling point of view) approach when using DB/2 PureXML. DB/2 PureXML and XPages are actually a match made in heaven (or was that the other place starting with h?), which is the subject to another story, another time.
Remember (before you start bitching about parent-child relations): the explanations above are guidelines, there are valid cases where RDBMS like separation of properties and detail rows into separate documents is a better approach. You know how to normalize, so I don't need to explain that part

A Document can save directly to an RDBMS

There was an ill-fated attempt to use DB/2 as storage destination for NSF data called NSFDB2 which unfortunately predated PureXML capabilities (otherwise the story might have been different), and it shall not be spoken about it anymore (support for it has ceased anyway). The tools here, available since R5 are DECS (Domino Enterprise Connection Service, included on every Domino server and LEI (LotusIBM Enterprise Integrator, an IBM product you need to buy). Filling in some simple forms in the configuration database can link your Notes forms (and partially views) to backend data from various databases or ERP systems. Once you got a grasp of these tools, some creative usage patterns become possible (more details here about Domino and RDBMS). Of course you can use code (ODBC, JDBC or LCLSX) your data transfer or use the IBM Swiss Data Knife for moving data around.

Next up: Domino views are different

23/12/2013

Domino Development - Back to Basics - Part 2: Forms and Documents

Category  
Continuing from Part 1 you now know that everything in Notes is stored in a Note. To further understand how Notes "ticks" some light needs to be shed on the relation between forms and documents.
In the simplest way to look at it, a form acts like the schema to be used to populate a document with values. The fields one fills into a form get stored as items into a document. This explanation is good enough for starters, but warrants a closer look to fully appreciate Notes' flexibility. In a RDBMS a record is "unbreakable" linked to the containing table by the database schema definition. Each record has each column defined in the schema. Notes ticks differently. The link between a form and a document is established by a Notes item with the reserved name "Form". When you open a form, fill it in and save it, automatically a item with the name of the form gets created. When that document gets opened, it will pull that form for display.
That link is not static: it can be overwritten in many ways:
  • In the Notes client using View - Switch Form ...
  • Though a field with the name form containing a formula
  • Code in an event
  • Code in an agent
  • A form formula in a view
  • A Smart button
A form ultimately only decides what gets displayed or computed when you open a document. Adding or removing a field from a form doesn't add or remove an item from a document! A popular trap: You add a new field to a form (e.g. color) and give that field a default value (e.g. green). You open that document to check and it nicely shows "green". However the document actually doesn't contain the item. Since the item wasn't there when you open it, the default value executes, but until you switch to edit mode and hit save nothing gets saved back into the document.
The same applies when you remove a field from a document: it simply isn't shown anymore, but the data is still in the document. If you want to get rid of the data too, you need to create an agent for that (which is pretty simple: FIELD obsoleteField := @DeleteField (You get the irony here: despite deleting an item the formula is @Deletefield. Renaming a field is the same as deleting and recreating it with a new name. It is an action in the form, with no impact on the document.
Another stumbling block: the form defines how an item is displayed and entered into a form via the field definition, suggesting data type and if it is one or more values. However items are always multi-value data and the data type of an item doesn't need to match the type of the field (if auto-conversion fails you will get an error). There is a detailed explanation about forms, documents and items you want to read.

Forms might include subforms, which from a document perspective doesn't matter. An item created (through a form) doesn't carry any information how it was created. If you need a detailed track record what item has been created/updated when and by whom, you want to have a look at the Inversion of Logging pattern.
Forms also contain a series of events that fire when things happen in the form. In the context of XPages that is no longer relevant, the event model is superseded by the XPages event model. However one area might be relevant: The @Formula used for fields:
How fields are computed
* Keywords: check boxes, radio buttons, drop down list etc. Depending on type are inherently single or multi-value or both
Fields in forms can be computed, carrying one @Formula or Input fields, carrying 0-3 @Formula. A field defined as computed for display doesn't get stored back into the document. A field computed when composed gets computed once when the document doesn't contain the field yet. Usually that applies to new document, but the formula also would run when a field is added to a form and a document is opened with that form (and if the document isn't saved, the result of that formula doesn't get stored back). A computed field gets refreshed every time a document is recomputed, that implies the document is in edit mode (unless there is no item with that name, then it runs in view mode too)!
The translation formula replaces the user input with its result. A popular prank is to pick a formula that computes something independent from the input. For orderly developers there is @ThisValue to use the user input. The Validation formula needs to return @success or @Failure("message"). The failure message gets propagated to the XPages error system and shows in the form error control, but not in a field error control.
In XPages a developer can define for each document data source that the document should be computed on open and/or save. The difference to classic form use: in XPages on open all formulas run: default, translation and validation while in classic on open only default runs. Also the form computation runs after the XPages validations, so if an XPages validation fails, the form computation never runs which then might yield additional errors from the validation formula.
Next up: Not all Documents are created equally

23/12/2013

Domino Development - Back to Basics - Part 1: The NSF

Category  
Over the last five years I have trained many developers on XPages. A good portion of them came from a non-Notes development background: Java, dotNet, PHP and others. Most of them had a better grasp on the necessities of web development than most of their die-hard-is-there-anything-other-than-the-Notes-client LotusScript colleagues. However they struggled to fully appreciate the finer points of the NSF based nature of Domino development. The little article series is for them.
In short: Notes is a distributed application platform with an integrated NoSQL document database featuring a rich security and event model. It can be accessed in many different languages and protocols. While it is written in C, the most prominent languages used to develop applications in Notes are LotusScript (a Basic dialect, now de emphasised), JavaScript and Java. The most used protocols are HTTP(s), NRPC and SMTP.

In the beginning was the NSF

Almost everything in Notes is stored in NSF databases (NSF = Notes Storage Facility).
The core structure of a Notes database
A NSF contains properties, an Access Control List, View indexes and a collection of Note elements, hence the name Note(s). The file name is not a property stored inside the NSF, however the title and the ReplicaID are. The ReplicaID has a special meaning: two databases sharing the same ReplicaID can synchronise their content in a process called replication, typically between a client and a server or multiple servers. When a server contains two or more NSF with the same replicaID replication can happen with any of them. A typical mistake: copy a NSF on the file system "as backup" creating not a copy but a replica. There is a menu action that creates a real copy for that (File - Application - New Copy).
Inside the NSF all data is stored in Note elements: both the design of the database, as well as the actual content. A data note (a.k.a Document) can be listed in a view (with the exception of a profile document), while a design note lists only in Domino Designer (unless you hack it). Design and data note elements share the same inner structure.
NSF do not contain any schema or data definitions. It is all Note and Item.

A Note is where you find anything

An individual note contains a series of properties and a collection of items (not to confuse with a field in an RDBMS or a field in a form). Besides data values for creation and update a note carries a UniversalId, a NoteId and a serial number. The NoteId is unique inside one specific NSF, the UniversalId is the identifier to link two note elements in two NSF (that have the same ReplicaId) together, so they synchronise on replication. The UniversalId is unique inside the NSF. Two different NSF can have documents with the same UniversalId. This is not an exception, but happens e.g. on mail routing. The serial number counts the times that note has been updated and is used in conflict resolution while replicating.
the basic unit of storage is a Note
The contained items follow the same pattern: they have a name, a set of properties and a collection of values. Two important properties are name and type. The name of an item doesn't need to be unique. Quite regular, especially when looking at the type RichText you have more than one item with the same name (for RichText the unwritten convention is to use the name Body). Items that start with the $ sign are used for system properties (e.g. $HasNativeMime indicates that the note contains a MIME document rather than RichText). Items also contain a serial number for conflict resolution in replication. The values in an item share the same data type. Items are always multi-value (a Vector in Java) even if the count is just one.
Next up: Forms and Documents

22/12/2013

Domino Development - Back to Basics - Overview

Category  
XPages development uses well established standards: JSF, JavaScript, HTML, CSS, JSON, HTTP and XML.
However it is different from other web development environments:
  • It is highly integrated. database, application and web tier come in one dense package. If your server is up Domino is good to go (no JDBC database timeout)
  • The native database is NoSQL (actually the mother-of-all-noSQL), a document oriented store
  • It is designed for distributed, partially connected operation
  • Code can run on a server or on a client
  • Domino provides build in directory and messaging capabilities
  • It provides a hierarchical, declarative security model
To ease new developers into "how Notes is different" I will publish a series of articles, that focus on the big picture. They will make it easier to avoid "coding against the grain" and to appreciate the power of Domino. The plan so far:
  1. The NSF
  2. Forms and Documents
  3. Not all Documents are created equally
  4. Domino views are different
  5. Finding data - Collections and Search
  6. Better save than sorry - Security
  7. Map Reduce Domino Style
Stay tuned

03/11/2013

Driving Embedded Experiences Adoption

Category  
With the introduction of Connections Mail and IBM Notes 9.0 a brand new productivity feature was made available in eMail: Embedded Experiences (EE). They are defined by the Open Social Foundation and can also be found in IBM Connections or Atlassian's Confluence Wiki.
The use case for embedded experiences in eMail is simple:
"Any application that sends a notification message to your attention and/or action can reduce the time and clicks required for processing by taking advantage of an embedded experience"
Ryan described in detail how to switch on the processing of embedded experiences and the Wiki tells how how to use XPages to send them. The trouble with this: you need to touch every single application that sends out a notification. You might not have time, budget or source code access to them.
So are EE a nice idea, but confined to some niche applications?
Not at all! When peeking under the hood, you can see, that an EE enabled eMail has a HTML and a JSON Mime part. If a client doesn't understand EE (like mobile clients or any other eMail than IBM Notes), then the HTML message is rendered. Important here: even with Notes you must have a HTML part, a plain text message won't do. Using a "when new mail arrives" agent, you can turn an incoming message into an EE.
As long as the target system supports one of the authentications the Account API provides (Basic HTTP, JEE Form, SAML, Spengo, OS Credential, Siteminder, LTPA) this will work. This is Notes client only, for Connections or iNotes you are limited to SAML.
First thought was to use a "Before mail arrives" agent to make sure, that the mail gets delivered fully converted. However such an agent runs in the router context and might be a performance killer - the when new mail arrives option seems a suitable compromise. The approach works with signed eMails too, since signing covers the eMail body, but not headers and mime separators (you can't sign the header since each routing hop adds information there).
As an example I used a plain text eMail, that sends a notification with a number of field/values separated by colon and containing only one URL. For your use cases, you will need to adopt the code (or wait for a future blog, just saying). Since my original message only contained text/plain I had to add text/html and application/json mime parts.

20/08/2013

Extend the Replicator

QuickImage Category  
One elegant way to improve perceived performance is to run computing task outside of user time. In Notes that is done using (scheduled) agents and scheduled replication (On mobile devices it is called PushMail ). When you have longer running tasks that only make sense when new data might have arrived, a scheduled agent doesn't make much sense.
Triggering a process "On Replication" is much preferable. Classic Notes agents don't have this ability, but the Notes full client can do that. Already today the replicator hosts not only classic Notes NSF replication, but also address sync and Activities sync. This type of extensions can easily be created using an Eclipse extension point:
  1. <extension point="com.ibm.notes.client.notesSync">
  2. <unit class="com.notessensei.demo.RunOnReplication"
  3. id="com.notessensei.demo.runonreplication"
  4. image="icons/replicate.gif"
  5. label="NotesSensei's Demo Replication">
  6. </unit>
  7. </extension>
There are a few caveats (besides designing an 32x32 icon) when building the class, but once you know them it is straight forward:
  • You either extend org.eclipse.core.runtime.jobs.Job or com.ibm.notes.java.api.util.NotesSessionJob
  • For both approached you must have a default constructor without parameter. Neither of classes you extend has one, but your implementation needs one
  • When subclassing Job you extend run() for NotesSessionJob you extend runInNotesThread(). The later has the advantage to have all the session handling completed, but you end with a dependency on Notes client code
  • While both methods provide an IProgressMonitor as parameter, your code needs to try to obtain the Notes specific one (see sample)
  • Keeping track of units of work is a pain, expecially when your tasks calls various modules with unknown number of steps. Luckily there is SubMonitor that allows to specify a set amount of "ticks" for a subtask that can then adjust them as needed (see example)
  • You should check IProgressMonitor.isCanceled() inside your code and terminate processing at the first possible point (without sacrificing data integrity) when encountered
Your class could look like this:

27/02/2013

Notes 9.0 search improvement

Category
The IBM Notes 9.0 availability anouncement is not far off, so the Notes user population, current and future can prepare for improvements to come. One of my personal favorites is the improvement in search, that now allows you to search in your current mailbox as well as any of your archives in one go:
search everywhere
Next stop (?): search in multiple|all local databases. But that's a story for a custom extension.

22/01/2013

What happened when - the Notes 1-9 time line

Category
The history of Lotus IBM Notes makes an interesting read (there's a Wikipedia version too). Since 1989 (that's 24 years) Notes has delivered releases that are fiercely backwards compatible™. I loaded the nifty fifty into a current Notes client and the R2 databases worked (after a compact) just fine. I like to put things into perspective:
Notes from version 1.0 to 9.0
There are a few factoids that are quite interesting:
  • Linux is almost as old as Notes
  • The first public release of MS Exchange was 4.0 when Notes released 4.5. Ein Schelm wer einen Zufall vermutet
  • Symbian predated Blackberry by 5 years
  • Sharepoint was introduced in the same year as IE6, could there be coincidences to the pain level?
  • Android was released in the same year as the iPhone, but it took a year before phones were available
  • There seems to be a huge period of inactivity after the 5.0 release.
    However there were not less than 12 point releases of the 5.0 code stream (5.0.1 - 5.0.13) with 5.0.8 the most popular (as I recall).
  • The biggest expansion of Notes were the following versions 6.0 / 6.5. So if history, as common saying goes, repeats itself, we are in for interesting days for Notes 9.0
  • 2004/2005 was a big year for Internet technology: the term Ajax was coined, Ubuntu and Firefox had their 1.0 release
  • There seems to be a huge gap between 8.5 and 9.0 of 5 years. However in each of this years IBM delivered a point release with new functionality (just some highlights listed, check the full release notes to see all):
    • 2009 8.5.1: XPiNC: XPages in the Notes client
    • 2010 8.5.2: Managed replicas, private iCalendar feeds
    • 2011 8.5.3: Performance, Security and Integration improvements (with fixpacks in 2012)
    Of course that version policy has a dark side: business users typically don't care for the numbers behind the dot and perceived Notes as "in maintenance mode" while it is alive and kicking
There are exiting times ahead for IBM Notes. See you in Orlando. Say hi.

31/12/2012

Avoiding login prompts in mobile approvals

QuickImage Category   
A customer posted an interesting question: "We send eMail notifications in our workflow applications. Our users don't want to be password prompted when following that link from their mobile devices. What are my options?".
While the Notes client can handle automatic authentication (especially with embedded experiences), in iNotes LTPA has logged you in and on PC platforms Single SignOn is well established, mobile device are trickier.
The "big" solutions would entail some form of Mobile Device Management (MDM), but that's nothing you want to deploy just for one app in question. You do want to plan MDM, but that's a story for another time (IBM recommends to do that in the context of an overall Endpoint management plan).
I see different possible approaches to get around the password prompt:
  • Use a VPN:
    A good VPN server can communicate with the reverse proxy and provide an LTPA token automatically. Sample code is available for Big-IP F5 and LotusIBM Mobile Connect. Implementing it for other VPN/Reverse Proxy combinations should be possible - check out Puakma SSO and talk to webWise
  • Use X509:
    After you deploy X509 certificates onto Android or iOS you can set the Domino Internet site document for your application to require X509 authentication. Since the certs are deployed on the device no additional prompt is required (of course that depends on how you secured the certs)
  • Go native:
    In a native (or almost native) application you can locally store the access credentials. You read/write data via JSON and https calls. Not too far off: use OAuth to authorise your mobile app.
  • Update (thx Mark, Per): Use OpenNTF's "Auto Login" project
I like the approach using a VPN with LTPA generation best since it saves you the trouble of managing the X509 certificates and adds a security layer on top
As usual YMMV

14/12/2012

Now that you can have embedded experiences in Notes, you need to send them

QuickImage Category   
Courtesy of Apache Shindig IBM Notes 9.0 (the client) and IBM Domino 9.0 (the server) now can render OpenSocial Embedded Experiences.
While Notes always had the option for custom mail experiences (store form in document, send), the Embedded Experiences allow integration into any application (that supports them).
One of the first things you might want to do is to pimp your existing applications to provide this new experience. It would be much lighter than storing the form and also work with other mail front-ends.
The good news: adding embedded experiences degrades gracefully, so all users not on 9.0 or later won't see a thing. This means you can get started immediately:
  1. listen to Niklas' explanation and get the sample code
  2. create/configure the gadget for your application (that's a separate post, coming soon), check the documentation
  3. create a class that does all the notification sending, classic and EE
To give you a head start for #3 use this.
As ususal YMMV

28/10/2012

The AppStore Conundrum

Category  
Legend has it, that Alistair, at a certain IBM event, challenged Nathan to saw controversy among the assembled participants. Nathan complied and succeeded by mumbling "We need an AppStore". So far the legend, we will never know, did this happen that way. Nevertheless the question remains:
"Does Notes need an AppStore?"
After all Apple has one for both OS/X and iOS, Microsoft has one for all flavours of Windows 8, Nokia has one for Symbian, as have Amazon and Apple for Android. There are AppStores for Firefox and Chrome. Clearly Notes need one too! There are attempts around already. Not so fast!

When looking at the AppStores one can see a pattern:
  • Applications are installed on local devices (or browsers)
  • If a server component is involved, it is provided in the cloud (outside the enterprises' firewalls)
  • The bulk of applications are for personal and social use
  • There are 10 to 100 thousands of applications
  • Most of them are free or low priced, designed for volume markets
  • AppStores mostly created a market, where there was none before
So how does Notes compare? With a few exceptions Notes applications are for collaborative use inside a corporate firewall and require a server component. A lot of Notes application are bespoke. IBM claims about 140++ M licenses sold, which is a way smaller target market than a whole OS or browser platform, who except the existing players would come along?
So there is no new market to be created. Administrators jealously guard their servers (not discussing the merits of that here). So all things a AppStore does enable more or less doesn't apply to Notes. Case closed! Not so fast!

There is room for improvement how applications are handled on the Notes platform. Two examples:
  1. In the Notes client I can hit Ctrl+N pick a template with a one line explanation, pick a server and arbitrary file name and get greeted with an error: "Bad boy, don't do this - anyway we won't let you"
  2. Have you tried to install the Connections Files Plug-in? Once you found it, download, expand, hack the plugin-customization.ini, install (Yes the admin could do that, but it is a drag
So my simple wishlist would be:
  • A new Notes.ini parameter NewDatabaseDialog=<some notes url here> and an IBM sample implementation for it (yes a more modern implementation of this). It would allow new predefined applications to be deployed more quickly than any manual process. Some accountability and "do you still use this" checks can be build in at gusto of the customer processes. Eventually a sync option with a vendor repository could be build in
  • A default toolbox.nsf created on each server install or upgrade. The database would have the extended functionality to be able to host (as a proxy) entries from remote repositories (e.g. by IBM, OpenNTF or our ISVs) and periodically check for updates (and download them for staging - think a little like RPM or DEB management in Linux)
  • A reliable automated patch management for the Domino server (but that's a slightly different story)
  • A revised home page for browser access, running on top of the application that provides the dialog of my first suggestion. It would provide access to the existing applications and the mechanism to request new ones
So users wouldn't interact with an AppStore in the wild, but the process of getting apps and tools deployed and keeping them current would be much more streamlined. Once all of that is in place and starts to make a difference I would revisit the AppStore question.
What's your take?

15/06/2012

How does an ideal Notes Client deployment look like?

QuickImage Category   
In recent customer discussions the question popped up: "How does the ideal Notes Client deployment look like". The intention of the question was less around the technical aspects, but the user experience. Technically you would have a shared install, a good widget catalog, some sensible policies, automatic login and continious defragmentation in place.
The question is: what should the user see?
There are components in Greenhouse and on OpenNTF. You also can roll your own from mobile websites or Java Code. You want to understand some RegEx. Don't expect your users to do that.
Chris Miller published his top 10, here comes mine. I distinguish between "official IBM", "community supported" and "custom stuff around LiveText". Here you go:
  • Official IBM Plug-ins

    • The build in Sametime client. While I have little use for the Buddy list, it provides the presence awareness for all Notes applications
    • The build in Activity plug-in. It allows me to keep my IBM Connections Activities offline, make sure to install them
    • The Connections Files plug-in: Seamlessly work with your files from connections, drag & drop files and links to files
    • The Notes learning widget: Have learning information at your fingertips
  • Community provided

    • Wildfire: Update all your status information in Connections, Twitter, Facebook, Sametime etc... from one convenient location
    • Bob Balfe's Attachment viewer: Preview attachments in the sidebar
    • The file navigator: Provides access to the file system, drag and drop like the Connections plug-in, but with local files
    • Discussion Add-on: One of my favorite "hidden gems". Copy any document (presumably eMails) with one click into a discussion database. Mini tool to enable sharing information with a team
    • Sender analytics: Find out more about an eMail sender (right click action)
    • Snippets: Single store for text snippets, attachments etc. Invaluable tool if you repeat with repeated information to different people
    • Connections alerts plugin: Stay in touch with the happenings on Connections while working in your Notes client
    • IBM community newsletter generator plug-in for IBM Connections: Generates newsletters and other interactions for your IBM Connections communities. Inbox still rules for notifications
    Not mentioned here: Sametime SUT, Softphone or Headset management - useful but vendor specific. Add them if you have them!
  • Custom configuration & 3rd Party

    The challenge here: find identifiable patterns that allow you to link text to external actions. Here are some examples:
    • \#([A-Za-z0-9_-]+)(?![A-Za-z0-9_\]-]): identifies Hashtags and provides the value with and without hashtag to search bookmarks or twitter or [insert-whatever-here]
    • \b([A-Z]{3}) ([0-9.,]+)\b: Match a currency (3 letter currency code, number behind)
    • /\b(1Z ?[0-9A-Z]{3} ?[0-9A-Z]{3} ?[0-9A-Z]{2} ?[0-9A-Z]{4} ?[0-9A-Z]{3} ?[0-9A-Z]|[\dT]\d\d\d ?\d\d\d\d ?\d\d\d)\b/i: Tracks UPS shipping numbers
    • TripIt: for the frequent traveller (add it as url based widget with authentication)
What am I missing here?
Is is quite some work to configure all of this, but is is worth the effort.
As usual YMMY

26/10/2011

Moving a NSF from one server to another

Category  
I will never FTP a NSF
And no copying with Explorer or SSH either. Moving from one server to another is called replication. If it doesn't work as you expected you have a problem at hand that you need to fix that moment or it will bite you soon enough! (Picture courtesy of Bart Simpson Generator)

11/10/2011

Filing eMail doesn't make you more productive - really?

QuickImage Category   
IBM research published an interesting study titled Am I wasting my time organizing email? A study of email refinding. It has been picked up in various places and looks like a field day for the "don't bother with organizing anything eMail" crowd. I think the paper is solid research for the question asked and the observations made.
However, I sense that a number of question haven't been asked that might change the producvtivity bottom line in favor of a different approach. I fully agree that hitting search or simply scrolling through a list of messages (often sorted by subject or sender) is a very efficient way to find a specific message. Also having a message thread is invaluable. I used message theads already in R7 when they only displayed when you open a message. So what did the study not look at (to be clear: I'm not saying they should have looked at these, but I'm saying you need to look at thse to gauge all over productivity):
  • Impact on productivity of a full inbox. How much time do I loose to every time I look at the inbox to decide: that's a message I dealt with, this needs my attention?
  • In Lotus Notes (other than Outlook) you can file a single message in as many folders as you like, so it is closer to tagging that "classical" folders. Once users know that, would they use it?
    E.g. I have a folder 1Action2Day and then customer folders. I move messages often into both. When I dealt with it I just remove it from 1Action2Day. Folders in Notes are equivalent to the tags in GMail, they could be a little more prominent in the mail UI (we do have the @Formula to show)
  • In Notes I can at any time scroll and search in the "All Documents", so I don't need to settle for one or the other strategy. Depending on my needs different strategies work better.
  • Difference in search strategy depending on short or long term tasks. E.g. I file "Thank you" eMails in a specific folder, so I can pull them out at review time. They are elusive to search since there are so many ways to say thank you (made my day, your'e the hero, awsome job etc). Also I use topic folders (e.g. a customer project) to review "what happened on the messaging wire". Using tags/folders I only need to make that association once when I process the message and not everytime when I search. So there's a huge difference between: "looking for a specific message" and "getting an overview what happened to {insert-filing-topic-here}"
  • Knowing that search is available, does "remove from folder" shorten filing time (I use that quite a bit)?
  • Impact of filing assistance tools. The paper mentioned one for Outlook and overlooked that we deliver one for Notes for a very long time: SwiftFile (unfortunately Windows only)
  • Impact of ad-hock filing strategy. I didn't really create labels/folder upfront but create them as I go. I also do not file everything. But I do remove everything from the inbox
The interesting pointers at the end of the paper can be translated: classifying messages is boring (true), automated tools can/should help (true), more people information might help to (good topic for a follow on research). I would conclude: good meta data would be best (aka "What is the context of this message"), if it wouldn't be so unconvenient to create them.

29/09/2011

Apples and Oranges Can I have 1785$ per user for messaging too please?

QuickImage Category  
Update: I promise: no more mental math at 1am in the morning. Post revised.
IBM invented FUD, but Microsoft turned it into an art. On their website they have an entertaining set of claims. I leave it up to you to judge their credibility. The real fun part is the Godiva case study. According to Microsoft the 1400 people company saved $250,000 per year by moving to Microsoft. That would turn into $178 per user/year. Of course that calculation is riddled with question marks. Lets remove the Oranges and compare Bananas with Bananas. There are 2 moves: one is from "I-run-my-own-server" to "happy-in-the-cloud", the other is switching products. So lets have a look:
  • When comparing deployment diagrams Exchange looks rather complex. The license stack doesn't look better. That might be one of the driving reasons for Microsoft to offer cloud. But what I wonder is how could they spend so much on running Domino? Back in 2002 Ferris conducted a study about Domino TCO. They found R5 would typically cost $22 user/month, while R6 would push that to $6 user/month. In R7 Ferris calculated additional 15% savings, with additional improvements with Notes 8.0 and Notes 8.5, especially around storage (DAOS), policies and monitoring. I checked with people who run efficient Domino installations and they run Domino, Sametime, Quickr, Protector at running cost of $6-7 user/month. So the number of products has increased and probablt the mailbox sizes, but no the cost. That leaves quite some money for licenses and hardware
  • But we are going cloud. So we compare to LotusLive Notes. The site says it is $5 user/month list price (or $3 if web only mail will do - like the factory floor workers). But let us us do more. Let's add file sharing (that also works with outside parties) and collaboration like IBM Activities or IBM Communities and Symphony live. This pushes the list price to $7 user/month. Finally we throw in web conferencing that allows to invite external guests for free. We end up at $10 user/month. That makes $120 user/year.
So when I shall save $178 user/year someone needs to pay me $58. (I know it isn't accurate, since there are people and servers etc. involved. But moving to LotusLive Notes is pretty much one cross cert, a few policies and then replication. So no cost for engaging and licensing BinaryTree). In German we call such claims "Milchmädchenrechnung". And we haven't looked at the apps yet.
Readying the flame proof underwear.
Update Thx for all the comments. Its good to know that there is some readership

30/12/2010

Simplify Domino installation on Ubuntu

QuickImage Category   
Installing Domino on Linux requires a few extra steps. You need to create user/group and implement a startup script. The best one can be found on Daniel Nash's site. A good instruction, step by step has been provided by Danilo Dellaquila in Part 1:Ubuntu installation and Part 2:Domino installation. Danilo went the extra mile and created a deb package for download that takes care of the details and simplifies your chores. Details are in his wiki.
As usual: YMMV

23/11/2010

Firefox takes a page from the Notes UI

Category  
For what it is worth: The iPhone made the tiled desktop, pioneered by Notes more than a decade ago, popular. Microsoft's ribbon was predated by a Lotus design. Now it's Firefox turn to adopt a Lotus UI element. The latest Firefox 4.0 beta release sports a menu button:
Firefox 4.0 menu button
That button looks very much like the Start button featured in Lotus Notes, Lotus Symphony, Lotus Expeditor and Websphere Portal
Notes Start button

27/09/2010

Lotus Greenhouse and the IBM OneUI v3

Category   
Have you paid a visit to Lotus Greenhouse lately? IBM features the upcoming Lotus Connections 3.0 there to try. There are a number of interesting observations to be made beyond the core improvements to Lotus Connections. Lotus Connections sports the upcoming IBM OneUI v3 and I expect the documentation being available at the usual spot once it is released. One interesting structure to note is the 4 basic tabs: Home, Profiles, Communities and Apps. Apps now hosts Bookmarks, Blogs, Files, Wikis, Activities and Forums (which makes the Latin speaker in me cringe: correct would be Fori, but that never made it into English grammar):
Lotus OneUI v3 Apps
There is no reason why that Apps menu couldn't host access to Quickr or your custom applications. So Lotus Connections evolves into the Lotus lead application. Looking at the OneUI you can see how it is evolving. This is one more reason to stick to OneUI for your custom applications. A simple CSS update will bring you in line with the new styles. Jeffrey Veen once taught me: Websites are about structure, layout and behaviour. Structure is provided by HTML, layout by CSS and behaviour by JavaScript. OneUI is all about structure - You don't need to agree with the CSS. Feel free to change anything inside the { }. Below you see the evolution from OneUI v2 to OneUI v3. Will we see user contributed OneUI themes anytime soon?
IBM OneUI v2
IBM OneUI v3
Can you spot the difference?
Update: The v3.0 documentation is out

06/07/2010

I did *NOT* create the AppStore wars video

Category  
Major blunder in This week in Lotus Episode 6. I have not been asked by Stuart McIntyre or Darren Duke regarding authorship of the AppStore war video. Nevertheless they seem to know that I created it. I did not. It is not my style of language and attributing me as the author will lead to serious trouble for me (there is something like IBM employee conduct guidelines that exclude this type of language for IBMers). I discovered it on YouTube (after being tipped off). Guys you owe me an apology.

29/06/2010

The AppStore wars

Category  
AppStores are the hot kids on the block. Everybody has one selling personal applications. Would that fly with Lotus Notes that is rather perceived as an enterprise application (is it?). Let's hear what the robots have to say:

Definitely someone needs to wash their mouth for using foul language.

UPDATE: NO NO NO. I did not create this spot.

07/04/2010

What if IBM would drop support for 32Bit Domino on Linux in favour of 64Bit?

Category  
Domino Server 8.5.x is available on a lot of platforms in a lot of versions:
  1. Linux for z/OS - 64 Bit
  2. Linux on Intel - 32 Bit (runs well on 64 Bit OS)
  3. Solaris - 32 Bit (runs well on 64 Bit OS)
  4. System i - 128 Bit
  5. System p - 64 Bit (32 if you ask very nicely)
  6. Windows Server 2003-2008 - 32 Bit (also can run on 64 Bit OS)
  7. Windows Server 2003-2008 - 64 Bit
What is obviously missing is a 64 Bit version for Linux on Intel (Linux here means: Redhat or Suse, Ubuntu isn't on the official server list [yet?]). Technically is isn't a problem since IBM has a 64Bit Linux version running. It boils down to customer demand and support cost. Every additional platform increases support effort (translated: fixes take longer). Occasionally IBM drops platforms from future versions of Domino (like: Netware, OS/2 or HP/UX).

So I'm wondering:
What would happen if IBM drops -in a future version- support for 32Bit Linux in favour of 64Bit Linux?
Would that be an issue in 201x? What's your take?

To make this crystal clear: This is me wondering, this is not any official IBM inquiry or a disclosure of any IBM plans.

24/03/2010

Fun with international versions

Category  
It is not enough to translate your software into another language.
Reply to me
You also need to teach it how to recognize Re: and Fwd: or you get the above result

30/11/2009

Domino Designer Import and Export

Category  
My colleague Guo Yi from the China Development Labs (CDL) in Beijing has published her first Domino Designer Plugin: Export and Import Tool for Domino Designer. I'm listed as contributor, which is true for idea and concept. All the heavy code lifting was done by Guo Yi. With this little gem you can export/import anything from/to Domino using the familiar Eclipse export/import dialog. The nice thing: the tool treats every file type appropriately: a CSS file becomes a CSS file and not "something like CSS wrapped into XML". A XPage ist XML following the XPages schema. Classic design elements are converted to/from DXL (so the usual caveats apply).
We were musing to add some kind of automation API to the plug-in (after all a version 1.0 needs a successor). ANT is my personal favorite, but we would be open to suggestion.

18/11/2009

Has replication as we know it reached the end of its usefulness?

QuickImage Category   
In a fully connected, always on, any device world there doesn't seem to be space for replication (or synchronization as it is called by other vendors). So why not scrap it and move on? I'm sure no one will miss the little diamond indicators denouncing a replication conflict.

14/08/2009

Software Design and Components in XPages

Category   
In Domino 8.5.1 we will have a new Design Element "Component". This design element can be used in Portal, Mashups and the Notes sidebar as iWidget as well as composite component in the Notes client. In a component you pick 3 XPages (could be the same for all 3 cases) for read, edit and help mode. You also define the events the components "knows" and what type of event is is for that component: sending, receiving or both. An event is associated with a data type that can be primitive (String, number), JSON or a complex data type you defined with that component.
Components are the Lego blocks of your applications. A system designer now can quickly build "empty" components wire them up and then leave the implementation to the XPages development team. This could accelerate Domino's RAD capabilities further.

19/05/2009

This morning @ NotesBench.org

Category  
NotesBench.org is the place where performance benchmarks for Lotus Domino servers are published. However this morning I was greeted in Japanese:
This
Which translates to: This is "notesbench.org".

We are ready, please wait.

05/05/2009

Example of a sidebar helper

Category  
Once you work for/with an international community or organization time zone pain becomes immediately a pressing problem. In Firefox I use the FoxClocks plug-in. In Notes 8.x I just got a new alternative for the sidebar:
When is it now
It is an IBM internal plug-in named "When-is-it-now" that not only shows you the current time at your selected locations, but also indicates with a traffic light how likely it is to reach someone based on business hours. Nice touch. The brain decodes a color signal so much faster than the time mixed with the question "is this a business hour".

21/04/2009

Building a high performance Domino Server

QuickImage Category   
Domino can take huge user populations. To do this successfully all elements of a Domino server have to be considered carefully. Following the old insight "It is always the cable" you need to pay attention to the hardware layout. While you perfectly well can install a Domino server on a low-end laptop or a VM Image, it wouldn't give you the peak performance you are looking for. You rather want something looking like this:
Server layout for a high performance Domino server
Let us look at the details:
  • Disk layout
    • Operating system and Applications: This is your first RAID 1 Array. Since data hardly change and are really not that much a small but fast spinning drive will do. RAID1 protects you against failure of one drive and speeds read operations. Some suggest to have separate drives for application and OS, but that might be overkill. You could consider having separate partitions (easy on Linux/Unix).
    • View Rebuild Directory: There is a nice notes.ini variable View_Rebuild_Dir. You can point to a separate drive to store the temporary files created during index updates. The default is the system temp directory. This directory is a good candidate for a RAM disk or a solid state disk when your system is updating a lot of views all the time.
    • Domino Data: Typically you have a RAID5/RAID10 storage here to accommodate the large amount of data (users demand Google size mailboxes and your applications don't shrink magically). More and more we do see SAN systems for Domino storage, which is OK. Just keep in mind: Don't store Domino cluster databases from different clusters in the same SAN since it defeats the idea of a share-nothing cluster. While we support the use of NAS, the network latency and bandwidth is a limiting factor. Archival servers run fine with NAS, but not your high performance primary production server.
      Update: Fixed the graphic to show RAID10 since is shows much better performance than RAID5
    • Transaction Logging: You have tried it. Switched it on, expected great things and it didn't perform. The flaw: for good transaction logging performance you need your own disk. Not just another partition, but your very own spindle (RAID1) ideally with its own controller. It would be interesting to see how solid state disks work here.
    • Full Text Index: Since Domino 8.5.3 you can move the FTIndex to a different drive. This improves data throughput and reduces fragmentation on your data drive. Add FTBasePath=d:\full_text to the notes.ini and run updall -f. Your 100 user server won't notice. Large environments will benefit
  • Network layout
    • Cluster Replication (If you cluster your server only): You want to have your cluster on its own network segment. If you have 2 boxes next to each other a cross-over cable would do (afaik 1GB Ethernet requires a hub). If your go three-way (highly recommended), then a hub and an IP address segment that doesn't get routed will do.
    • Server Network: All servers should be connected on the server backbone. Put them into their own subnet clients can't see. Replication never gets disrupted by clients jamming the network ports. The server network also handles mail routing.
    • Client access: If you have huge numbers of clients you might reach the physical capability of your network card or the TCP/IP stack. Use more than one card and/or more than one IP address to have sufficient ports available for clients to connect.
Of course all of this isn't new (except the shiny picture), you can read much more details on IBM's Domino Performance Best Practices pages. This is just about the hardware layout. You need to consider the operating system too. But that's a story for another time. As usual YMMV.
Update: There is now addtional material available how to tune an IBM System x server to peak performance. Update 2: Samir points to a nice comparison between RAID5 and RAID10. It's not Domino related but insightful. One key point there: watch your controller.
Update 3: Added the separate drive for the full-text index

04/04/2009

Accessing Sametime Unyte from your Notes Client

Category   
There is a nice little plug-in available to instantly host or join Sametime Unyte meetings from the comfort of your Notes 8.x or Sametime 7.5x++ client.

Installation instructions can be found @ IBM Support.

02/04/2009

IBM Lotus Domino 8.5 performance for iNotes users

Category  
The second performance report is out: iNotes 8.5. From 8.0 to 8.5 alone disk operations/sec improved between 22% and 33% and disk bytes/sec between 24% and 45%. Improvements from previous version should be even bigger.

01/04/2009

LSX Toolkit finally gets updated

Category  
After a long wait the new version of the LSX toolkit is available. There is great rejoicing in many places soon after Brent announced it. And life is good.

13/03/2009

IBM Lotus Domino 8.5 performance for IBM Lotus Notes users

Category  
The Notes 8.5 mail server benchmarks are out.
From the article: "Increasing value to the customer is an ongoing theme for the IBM?? Lotus?? Domino?? server. The growing volume of information in today???s enterprise strains existing hardware infrastructures. Lotus Domino 8.5 contains features that help relieve the stress on the expensive processor and storage subsystems. This article shows the reductions in processor and disk utilization with Lotus Domino 8.5 and Lotus Notes?? clients."
I spoke to Yang Bin and he confirmed that the iNotes performance paper will follow soon.

24/02/2009

Hidden IBM Software Gems

Category  
When looking into the software universe you usually find 2 kinds of software: standard software (like OpenOffice, SAP, Lotus Notes) and custom (or bespoke) software (which is build by an ISV or an inhouse team). On a closer look there is a third category. In IBM we call that a software asset. Such an asset solves a business problem in a repeatable way but hasn't enough breadth or depth to justify the process that turns it into a standard software. Typically software assets are used together with consulting and implementation service. Inside Lotus, to no surprise, there are a lot of these assets under management by the IBM Software Services for Lotus (ISSL). From time to time I'd like to introduce this solutions. Here are two:
  • Automated Database Deployment for Lotus Notes(ADD4LN - IBM internally known as Database-o-matic): Allow business users to create new databases based on selected templates and manage them including the access control. Users won't need to learn the ACL dialogs or the Notes admin client.
    Why is it interesting: One of Lotus Notes' biggest fans clearly stated, that databases are the key to Notes bliss. However allowing anybody create access to any server only ends in a maintenance nightmare. ADD4LN fills that gap by not only making creation easy, but also helping to keep track. ADD4N does this one thing very well, and is fast and easy to implement. If you are looking for a more versatile solution, you need to ask Bill.
  • My Attachments Tool (MAT). MAT moves automatically or on trigger all attachments from your mail file into a separate attachment database and replaces the original attachment with a DocLink to the new location. It identifies duplicate files and only keeps one copy. MAT predates tools like DAOS and Quickr and is still a valid addition to your toolset. MAT works on Windows, Linux, Mac and OS/2. Normally you would use a local attachment database, so no server side storage is required. I have configured my MAT to archive any attachment that is older than 7 days.
    Why is it interesting: While that 50 MB Powerpoint presentation is stored only once in DAOS for all 500 users, it still counts 500x against the quota. With MAT you can remove these attachments from your quota and still enjoy the fulltext search and intact context of the attachment (who send what file when is often as important as the content itself). MAT works offline by design and can be replicated if desired (the Quickr connectors are not there yet). MAT also helps to eliminate local duplicates. I use it every day (OK, the background agent works for me every day).
Your local IBM sales rep will know more - or drop me a note.

22/02/2009

Keeping track of events

Category  
Every large or small organization has events that everybody needs to be aware of - as needed. There are many solutions that offer corporate and team calendaring. They all have a drawback: you need buy them and you need to change the way you work (Some are smarter than others). Today I'd like to show you one option how to get a corporate calendar on a shoe string (You need your admin's help for that):
  1. Create a new database on the server using mail[yourServerVersion].ntf as template
  2. Set the default access to -No access- and [x] Read public documents (don't forget to have a manager access for the database)
  3. (Optional): Add a group with -Author access- and [x] Write public documents. They could create new events in that database
  4. Go into the profile and set the Owner of the mail file to Corporate Calendar/MyCorp
  5. In the calendar profile set the autoprocessing option to "accept initiations even with time conflicts
  6. Define a mail-in database in the Domino directory with the name Corporate Calendar/MyCorp pointing to your new database
  7. From now on for all events you want to be visible, enter them as meetings and invite Corporate Calendar/MyCorp
  8. (Optional): In Notes 8.5 add this calendar to your calendar overlays.
Of course you can have many calendars this way: by department, internal and external events etc. Short of the calendar overlay this possibility has been around since Notes R4.5 when calendars have been introduced into Lotus Notes. Hint for IBMers: you also can use the calendar of a functional ID for this purpose. Getting one of these is a well established process.

18/02/2009

Domino eMail for Android devices (and others)

Category  
On 16 February Seven announced at the GSMA Mobile World Congress in Barcelona the availability of their email solution for Android phones. From the press release:... Support for business email: provides mobile subscribers with seamless and secure access to Microsoft Exchange or IBM Lotus Domino corporate email, which represents 80% of the corporate email services in use without installing any software behind the firewall....
Seven already supports Brew, J2ME, Palm, Symbian and Windows Mobile. Seven is available in more than 50 countries.

07/02/2009

The nifty-fifty unearthed

Category  
Catching up on my Blog reading I came across a post from Tony Austin ranting about rants about Notes 9.0. In that post is a gold nugget (cat's gold I suppose): a link where you can download the original nifty-fifty (for the uninitiated: the nifty-fifty were 50 sample applications Lotus published ca 1999 1992/93 together with Notes R3/R4 in the early days of collaboration, where people would use groupware for a wide range of tasks and hadn't turned into eMail junkies). Looking at these applications I can't starve off that feeling that all the new web2.0 hype is pretty much the same old wine in, admitting very pretty, new barrels. These are the applications:
  1. Things to Do
  2. Requisition Approvals
  3. Application Tracking
  4. Book Reviews
  5. Budget Planning
  6. Call Tracking
  7. Clip Art Gallery
  8. Contract Library
  9. Correspondence
  10. Customer Service (five databases -- Product Design Problems, Customer Suggestions, Knowledge Base, Data Dictionary)
  11. Call Tracking -- as well as a sixth database with documentation)
  12. Demo Room Reservation
  13. Employee Records
  14. Event Calendar
  15. Executive Briefings
  16. Expenses Reports
  17. Formula Catalog
  18. Forms Routing
  19. Electronic Library
  20. Icon Catalog
  21. Job Candidates
  22. Job Posting
  23. Sales Management
  24. Library Tracking & Training
  25. Purchasing Item Tracking
  26. Litigation
  27. Lookup Keyword Library
  28. Meeting Tracking
  29. Monthly Financials
  30. Engineering Notebook
  31. Patient Treatment Signoff
  32. People and Facilities
  33. Phone Book and Yellow Pages
  34. Presentations
  35. Race Results
  36. Real Estate Management
  37. Recipes
  38. Relocation Expenses
  39. Routing Tracking
  40. Reservation Scheduler
  41. Sell-through Sales Reports
  42. Shotgun E-Mail Communications
  43. Conference Suggestions
  44. Support Conference
  45. Support Center Call Tracking
  46. System Equipment Inventory
  47. Team Consolidate
  48. Team Room
  49. Tech Services Call Tracking
  50. Technotes and Tips
  51. Telemarketing
  52. Travel Authorization
  53. Values and Goals
  54. Wholesale Customer Tracking
Tony of NotesTracker fame lists 3 locations (2 of them his) where you can obtain a single zip file with all these databases: All of these applications are vintage and look like 1999, some are outdated but the rest could be a nice foundation for an XPages feast. Xifty-Xifty anone?
[Update]:The databases are in R2's ODS13 format, to save you the trouble of conversion they are now available in ODS51 of R8.5. Still the design hasn't changed, but there is hope (Do you now understand what a circulare reference is ).

03/02/2009

Protect your IBM Lotus Domino Server against brute force attacks on non HTTP ports

Category  
Chris Brandlehner provides excellent advise how to harden a Domino server further against dictionary attacks. Of course you only need that if you allow non-Notes clients to authenticate via SMTP. If that's off (as in good Notes/iNotes only shops), you don't need to protect that vector. Anyhow the script is designed for Linux, I guess it works on AIX, Solaris and other Unix too. I wonder what you do on Windows.

07/01/2009

Overheard at a developer discussion

QuickImage Category    
XPages feedback from an early adopter:"I'm now able to call our code from an XPage... and equivalent transactions that were taking a full second (or more) to complete in a Domino agent are now completing in 20 - 40 milliseconds. And we haven't even optimized it yet; that 50x performance boost is just from running it in XPages instead of in agents."
What do you want more?

23/12/2008

XPages Workskops coming up.

Category   
Tim sums it up:
So it seems that word is spreading about this great XPages stuff, (so someone does read my blog) and I am being asked what the plans are to deliver it in other countries. January is going to be planning month and Lotusphere 2009 So here is who to contact if you want to add you name to the list of people/countries that are interested.
to find out how to contact them you can use the WhoIs application.
Only Stephan and I know about this blog post so far so when you contact the others, please be gentle. ;o)
  • EMEA: Tim Clark Please leave comments on this post to show an interest for EMEA.
  • Asia Pacific: Qiao Ming Chen for China, Stephan Wissel (me) for Asian & India, Atsushi Sato for Japan
  • ANZ: Craig Hart for Australia / New Zealand
  • Americas: Craig Wolpert

23/12/2008

IdeaJam running within Lotus Connections 2.0

Category  
IBM's business partner Elgui Software has a great piece of web2.0 software called IdeaJam (build on Domino/XPages). IdeaJam allows the sharing of ideas, commenting and voting on all ideas. It is an application no IBM customer should miss. Bruce just pointed me to the latest feature: seamless integration into Lotus Connections.
From the IdeaJam product page: "IdeaJam is a place where people can post and share their ideas, and gauge the marketability, popularity and viability with input from others.
When an idea gets posted, others can help promote or demote the idea and provide comments. Popular ideas will bubble to the top.
Make your opinion count.
"
Go and check it out.

18/12/2008

The "real" impact of XPages

Category  
Based on the IBM Dublin workshop Tim,Matt and myself created the materials for the famous XPages workshop that ran in Hursley and Bejing (and is planned for other locations early 2009). One comment by a participant summed up the impact of XPages on Notes (yes it's coming for the client) and Domino:
"I was afraid that my lack of experience with the Lotus environment would prove to be a burden during the session, but it has not. On the contrary it might even have been an advantage (emphasis mine), having experienced the workshop with clear eyes (if you will) not trying to apply old routines. It has been difficult to accept drag & drop and the "what you see is what you get" concept of developing. Coming from normally developing in a simple text editor basically. But it's quick effective and if there's any need to goof around with the source, the option is there. I think I will definitely embrace XPages and hopefully with the knowledge I've gained this week, have my colleagues do the same. (They are all Lotus fanatics anyway :) )"
A lot of our customers cite difficulties to find enough Lotus developers. With XPages Domino you develop in JavaScript end-to-end. These skills are much more readily available. Of course you need to learn the object model, but at least you get away with a single language.

10/12/2008

Testing Notes Client applications

QuickImage Category   
My friend Lucius started his own software company Smart Toucan. His first product is Auto-User, a tool that can test Notes Classic client applications. You can use it for recording/replaying demos or to run automated software test. While IBM has a great product for web application testing (the Rational team will not talk to me anymore if I don't mention them here <g>), automated testing capabilities for the Notes classic client are painfully absent. Auto-User plugs this gap. The current Beta release works with Notes 7 and I'm looking forward to the R8 Classic and Basic configuration versions. Go check it out.

08/12/2008

Estimating Notes projects

QuickImage Category    
Estimating software project cost and duration seems to have parallels with the local weather reports: You simply can't rely on them. Waterfall project approaches are partly to blame. Estimating the amount of work units for a requirement is another. How much work is "The system needs to be user friendly"? For the sake of this post lets us presume your have a magic box, typically labeled experience, either explicit or implicit, that gives you an insight how much code you need. Once you know how much code (a.k.a. LOC or "lines of code") you need to write, estimation becomes manageable. A good tool to estimate is COCOMOII. With the LOC, COCOMOII and your average hourly cost you can estimate the cost easily (Thanks Mitch).
However using a visual development environment (Visual Studio, Eclipse, Adobe Studio, Domino Designer) makes things a little more complicated. A form or a view you draw is not a line of code. The UML model you paint (that will turn into a database schema and code classed via generation) is not a line of code. So you need to expand your mental estimation model slightly. There are two new terms to get familiar with:
  • LOC_TE = "Lines of Code - Time Equivalent": The time for painting one form is equivalent to writing how many lines of code?
  • LOC_AL = "Lines of Code - Avoided Lines": When you paint one form how many lines of code can you avoid to write? The whole promise of any visual environment is, that LOC_AL > LOC_TE. This probably varies from platform to platform and makes an excellent subject for a bar brawl between competing developer camps. I guess you won't find much information about that ratio since nobody ourdays would afford to build a system twice.
Where does that leave you for Notes or Domino estimations? You need to find your magic number for LOC_TE for the various design elements (see graphic below) and then apply the COCOMOII estimation. Of course you could simply use time estimates (by revisiting your past projects), then however you need to calculate your own project variance based on project complexity and team experience. Neglecting experience and complexity are the great fallacies happily endulged in by project planners with too little experience in software project estimations. The difference for a similar sized project can be 500% from "very experienced team working on a low complexity project" to "unexperienced team working on a high complexity project". Even if you just consider experience you will still have to shell out more than double the money (230%), 70% more people and 40% more calendar time for an inexperienced team.
NSFSoftwareEstimates.png
The orange boxes are most likely what you will get out of the discussion with the business users, including some of the yellow boxes. For the red ones you can directly insert lines of code. How do you estimate your projects?

05/12/2008

Notes Application Statistics

Category  
I recently had the opportunity to look deep into the many thousand applications one of our customers has deployed. I found the following metrics:
  • A Notes database application has per average 18.8 forms, 14.4 subforms and 37 views (including folders)
  • There are 8000 lines of code per application (@Formula, LotusScript, JavaScript, Java) and 10.5 agents
  • A form or subform has per average 10.7 fields and 8.8 actions or buttons. So if a form uses 3 subforms you would have 32.1 fields in that form.
  • A view has per average 6.7 columns and 8.8 actions (We averaged actions across forms/views, so the number is the same as for forms/subforms)
  • The Lotusscript functions have an average length of 21 lines
Of course: averages are averages. Typically you find a bunch of auxiliary forms/views with small numbers of actions/columns/fields and the main forms that tend to be much longer. Guess I'll play with that statistics a little more.
YMMV

17/11/2008

applicatio sine qua non*

Category  
If you only have money for one intranet application running on your Domino server (besides the blog and discussion that came with the server and all the goodies from OpenNTF) it would be IdeaJam from elguji Software llc. My favorite other Notes application (which runs on the client) also uses IdeaJam to collect ideas. No idea what ideaJam is? Just watch the video:

Ready to learn more? Take the tour. IdeaJam (as the time of this writing) is available in French, English, English (GB), Spanish, French, Italian and German. If you order enough copies they would make a Chinese or Bahasa version too.

* The application you can't be without.

16/11/2008

Lotus Notes Applications and other eMail Systems

Category  
Microsoft's competitive strategy with Lotus Notes is to migrate eMail to Exchange (more likely to Outlook with Exchange included as collateral damage) and to sunset or migrate applications thereafter. Looks good on paper and eMail is eMail isn't it? The hangover from the "Hooray we'll do Outlook" party comes when running the numbers on application migration. Remember: Every minute/dollar spent on migration doesn't get spent on new user requirements and productivity improvements. Their poster child Accenture, with Steve Balmer sitting on their board (would that eventually have influenced their migration decision?), took more than 6 years and still according to Vidya S. Byanna, Global Infrastructure Executive Director Accenture 200 business process supporting databases are still not migrated (nicely put by Vidya: "are in the process of migrating"). Update: Accenture has removed the blog entries, but
The web does not forget! Quite interesting finding given their access to Microsoft and their claimed technical expertise. So if you have started a migration, good luck to you. Running your numbers carefully, you might end retaining and extending Lotus Domino as your collaborative platform (There are a lot companies doing just that). eMail is sooo last century. Of course the question of interoperability needs to be answered. When you build web applications, you probably already wrote a class MailNotification that generates notification eMails with http hyperlinks in the message. If you have existing (client) applications you need to deal with the way @MailSend and NotesDocument.send is working:
  • You have retained your Notes clients: Don't do anything. Connect your other eMail system to the Domino system using SMTP and the SMTP router of Domino will convert the DocLink into a hyperlink using the notes:// protocol. Of course I presume you have configured your Domino server properly (hostname anyone). If your user typically use local replicated databases the links created will point to local databases using notes:///. That can be a problem if the receiving end doesn't use a local replica. In such cases use Geniisoft's CoexLinks.
  • You web enable your application using the Domino http task (classic or XPages alike): you either need to touch all your applications and replace @MailSend/NotesDocument.send or use Geniisoft's CoexLinks. CoexLinks is an unobtrusive server tasks that does link conversion. It also takes care if users on Lotus Notes send you a DocLink. And NO. There is no magic button for web enablement. Depending on the code quality and structure of your applications it can be very easy or a little painful. See my session at Lotusphere to see what could be automated.
  • Your messages use stored forms and that forms contain actions and buttons and the like: Sorry. You need to rework these parts.

03/11/2008

Domino 8.5, what's in the box?

Category  
A customer recently asked: "How difficult is it to migrate from Domino 8.0 to Domino 8.5". I gently had to remind him that we upgrade while others migrate. So what's the difference: A migration often involves new hardware, while a upgrade usually happens in-place (unless of course your box is rusty and you want a shiny new one). A upgrade doesn't alter data (a lot) while a migration typically requires data conversion. A upgrade can be rolled back very fast (in Domino: just start the server from the old binary directory - you might need a little help from the compact task beforehand). A upgrade also typically coexists very well with older releases. A migration comes with coexistence challenges.
Got it? Our colleagues from the Websphere brand came up with a very interesting model for new functionality. Instead of releasing a new version altogether they publish feature packs. Taking this train of though you could look at Domino 8.5 as a combination of a maintenance release and a feature pack. It will include more than 3000 fixes since the release of Domino 8.0 and a set of new features
domino85feature.jpg
  • DAOS which stores attachments one time per server regardless of the number of users it got sent to
  • ID Vault which help you to manage that Notes.ids once and for all (I haven't come across a better PKI store yet
  • XPages which allow you to do web2.0 with elegance and ease. XPages is actually not new technology. It has been available for Websphere Portal as Lotus Component Designer for quite a while. New is the deep integration in the Domino stack (and Domino Desginer, but this is a server post)
Of course you want to check the Official Pages once 8.5 has been released.

02/11/2008

The shadow of things to come

Category  
Lotus Notes and Domino 8.5 are closing in on their release date. There is a lot of buzz around XPages. But IBM doesn't stop here. Development for 8.5.1 is in full swing and planning for 9.0 has kicked off long ago. Of course (can you spell NDA) I can't tell what's happening next in the Domino space. But I can share a screenshot from my workstation (proudly running Ubuntu):
dominobuntu.png
Nota bene: I do not work in development, I don't make statements what they do or will do.

07/08/2008

Coming Soon

Category  
YellowDayTeaserSmall.png
Stay tuned!

01/08/2008

Spreading the love

Category   
To often people still think Lotus = eMail. That is worse than thinking Lotus Notes is eMail. Yes there was a time when Lotus = Spreadsheet. So what to do about it? Well a picture is worth as thousand words, so everytime I open my laptop the picture becomes clear:
notessensei_laptop.jpg
notessensei.com is an alias for this blog. Working on my Global Microbrand

Update: The skin has been produced by Digiskin. The pro: it is super-thin, the con: you can't fix it yourself, you have to go and see them. They seem to operate in Singapore, US, South-Africa and Germany.

09/07/2008

Domino and eDiscovery

Category  
Most legislation today require to keep "accounts and other business records" for many years (typically 3-8) for compliance and discovery. Depending on their content eMails are today considered business records*. When starting to look into eDiscovery there are a number of terms and concepts one needs to become familiar with: Sub Poena, Sarbanes-Oxley, Basel II or eDiscovery.
Once you wrapped your head around it you then need to architect a system that avoids massive violations of retention legislation. Having all data stored in a single shared database like MS Exchange will immediately lead to the need to buy add-on products. Domino on the other hand provides a lot of required functionality out-of-the box. Dan Lynch has summarized what you can do. Go read it.

* While an email with an invitation to an after-office beer most probably isn't a business records, however the invitation to lunch to discuss the proposal surly is a record since it documents part of the history how a sale closed.

26/06/2008

Excel to Notes or Quickr?

Category  
Question to the readers out there:
Has someone written a tool that allows to take a spreadsheet and turn it into a Quickr place or a Notes database. Of course that spreadsheet would contain a list. The tool would pick up the column headings turn them into field names and generate simple forms and views with the ability to do in-view-editing. I know a number of solutions that import/export spreadsheets but to my knowledge they all require existing forms and/or views. I admit I haven't done an intensive research on that yet.

Feedback very much apprechiated

12/06/2008

Domino does sync with the iPhone!

Category  
The heat is on. So what are the options for Notes and Domino users?
  • Use Domino 8.0.2 (expected in August) and iNotes Ultra light. iNotes UL will support the iPhone. While it isn't Pushmail is is painless and doesn't leave stuff on the iPhone to be wiped
  • Wait for IBM to licence Active Sync. We currently predict that for the moment hell freezes over.
  • Find someone to implement Apples native protocols (we suspect they are http/webDAV/CalDAV)... which are not documented (are they?)
  • Use a robust sync server that can talk to Notes and Domino
Yes, it is available. Funambol just announced their support for the iPhone. You can kill a number of birds with one stone since Funambol supports a wide range of devices. Go check them out:
(C) 2008 Funambol, I'm sure they don't mind
Did I mention that they are OpenSource?

03/05/2008

Brunei ICT Career Day

Category   
I was presenting on "Workforce 2.0" at the Brunei ICT Career Day. The event was organized by new newly formed InfoCom Federation of Brunei (IFB), the IT vendors industry association. I know the 2.0 memme is a bit overstretched, so I decided to present my case with a little tongue in cheek. In a nutshell: well trained talent is more mobile than ever, so attracting and retaining them requires enterprises (the 2.0 versions of course) to adopt to the work and collaboration style favored by them. They will blog and twitter about work and their customer interaction anyway, so providing them with the platform within makes that more valuable for their companies (Knowledge Management anyone). I got good questions from participants from AITI as well as the Prime Ministers Office: How do you "police" a Social Software environment against abuses: time wasted or even use against corporate interest. The example given was: If an IBM engineer uses the communities tool to gather a group of experts with the intention to jointly leave IBM and setup an independent, innovative and competing company. My reply to that: it happened before (SAP anyone?) even without these tools. Today IBM probably would have a close look and later buy the new company.
In the evening we had dinner with Bruneian customers and a good set of discussion items: How do you design a long term national development plan when the world is developing at "Internet speed"? Somehow inevitably the discussion later touched the topic of the recent Malaysian election, where the common understanding is, that bloggers played a big role in the mood swing that led to the heavy election losses of the current government. One of my Malaysian colleagues attributed the influence of the bloggers mainly to the failure of the Malaysian government to reach out to young voters and their failure to engage in dialogue to address the growing frustration about progress and inclusion (or the lack of) in Malaysian politics. As my mom always told me: you better listen!

02/05/2008

The shape of things to come

Category  
Alan reported about it first. Our Business Partner Pentos AG is integrating Skype into Sametime and Lotus Notes. What started as a "simple" plug-in rapidly evolves into a fully integrated solution. Tim (the lead developer) shared a preview of the next iteration of the plug-in with me. It shows the Skype status wherever you have a Sametime enabled field. How kewl is that.

Sametime and Skype

07/04/2008

How I got started with Lotus Notes

Category  
OK it is going round. So here is my story. After National Service I went to law school in Würzburg and Frankfurt. To make some money I worked as intern in IBM 1985-86 in my specialty, the IBM /36. IBM introduced me to the all new PC / PC XT and PC AT. Very soon I worked as freelance consultant and trainer for  applications like IBM Displaywrite, Harvard Graphics and Lotus 1-2-3. In a gig with Sears I learned the @formula language on the fly (the manual, me and the midnight oil) as well as the secrets of monthly financial reporting (I had some experience since I did COBOL at the university).
So I got pretty good at teaching and programming 1-2-3. I also used the later Lotus Products like Symphony, Magellan and got really fond of Lotus Agenda.
When Lotus Notes 1.0 came out, I was curious and got exited, so I called Lotus and wanted to buy Notes licences. I've been told they would only talk with me 500 licences onwards. So I looked around in my office and asked back, what I should do with the other 497 licences, so the deal was off.
After Notes 2.0 came out, in 1992, I tried again, getting the same answer but with the remark, that a Lotus Business Partner could sell me less licences. So I looked around and at the end bought my first Notes licence from Haus Weilgut. They are still around and are the makers of the Lotus Notes MindMapping application Mindplan.
After working a few month with Weilgut's CRM solution curiosity got the better of me and I fired up designer (which was part of the Notes client then) and started to mess around with the forms and views of the applications. Pretty much everything I know about development I learned in self study and by picking any brain that came close enough. 1996 I started to work as freelance trainer for the Digicomp AG in Switzerland and got my CLI certification. Being a CLI in Germany Lotus Education started to take care of me and I got quite busy. 1997 I met Bob Balaban at the Lotus Advisor Magazine Conference in Phoenix, AZ. I got a hand signed copy of "Programming Domino 4.6 with Java" and consider Bob a friend since then. 1998 Credit Suisse did send me to Singapore and Hong Kong for a training project and I decided to stay. After a few steps I signed up with IBM in 2006 as Lotus Technology and Productivity Advisor ( LTPA for short).
What's your story?

01/04/2008

Domino next++

Category  
Domino's next version is under active development and as much as I would like to share details I'm bound by an NDA. However glimpsing into the future beyond the current version is something I can do (to some extend). To understand where Domino is heading, you need to be clear, that software is about people. Always has been, always will be. A little story to illustrate: In the 70th and 80th IBM trained a lot of IT experts in internal training programs only to "loose" them to the open market soon after their training completed. Everybody thought IBM should bond their trainees, until it became clear, that the "lost" trainees turned out to be the most loyal customers.
So again " The emperor sends out his knights" to ensure the future success (OK, one came back already). With our experts in strategic positions big things are in store for Domino next++ (Could be Domino 9.0 or Domino 10.0).
The interest the DB/2 integration led to some radical ideas around the storage engine and I'm exited to share that Domino will support CouchDB as storage engine. Since Damien did wonders to the @Formula engine, he will do that again for the database. He will work with the soldiers in the field to also provide seamless connectors to access local stores of other messaging systems natively (so no more migration of your Outlook or Thunderbird archives, just use them).  The Domino web container will be powered by the Zero engine. Zero (together with Jazz) is my favorite IBM project. Zero provides language environments running on a JVM: JavaScript, Java, PHP. So the Domino implementation will add LotusScript.0. Just imagine the possibilities: Writing Domino agents in PHP, Execute an @Formula in Java without invoking the ORB. We have some exiting possibilities coming up.

05/03/2008

Lotusphere Comes To You Hong Kong

Category  
LCTYHK2008.jpg
A good crowd turned (my guess is between 250-350) up to get their dose of Lotusphere.

12/01/2008

Playing with BlogSphere Skins

Category  
My current past work chill-out activity is to play with Skins for Blogsphere. Declan made BlogSphere 3 compatible with Movable Type Themes, so you have access to a great library of design. Probably after a few rounds I'll get tired of switching and settle on one. Until then don't worry if you see a page that doesn't look familiar. And yes: I'll let Declan have them for the next release.

18/12/2007

Not your dad's gMail anymore...

Category  
We all love gMail (more or less) almost as much as Domino (and there is a Sparkle in the near future). So how about this:


Thanks Bruce - and yes also available in English.

16/12/2007

J2EE processing pattern

Category  
Lance has a good time in Singapore (OK, that thing about running in the morning is on his " learning-the-hard-way" list)
When discussing Project Blizz we realized, that we needed to discuss execution sequences. While all the modern tools ( MyEclipseIDE, RAD etc.) provide drawing tools, they are not suitable in a discussion where the flow changes rapidly. Nothing can beat a white board there, as long as the team is in one place. Since Nithi had to return to India we had to look for something else. Luckily I found Sequence. This little gem translates pseudo code into sequence diagrams as you type. So instead of fiddling around with layout (I'm not on a Mac), one can just type some lines of text and get a diagram in an instant. Our pattern for processing Domino stuff through the Websphere layer looks like this:

ServletOrTag.getResult {
	stuffRenderer.render(configName, parameters) -> OutputStream {
		configFetcher.fetch(configName) -> configObject;
		stuffTransformer.setSession(dxp2Session);
		stuffTransformer.transform(configObject, parameters) -> OutputStream {
			stuffFilter.setSession(dxp2Session);
			stuffFilter.filter(configObject,parameters) -> OutputStream {
				stuffFetcher.setSession(dxp2Session);
				stuffFetcher.fetch(configObject,parameters) -> OutputStream;
			}
		}
	}
}


And Sequence turns that into this:
Sequence Diagram

09/12/2007

Lance - Welcome in Singapore!

Category  
Lance Spellman
Lance is joining me in Singapore Monday 13th December 2007. Welcome!

11/11/2007

Can someone explain where Exchange is better than Domino?

Category  
I don't get it.
There is a lot of discussion about what is better Microsoft Exchange or Lotus Domino. Looking at Exchange (of course with my "working for IBM" bias <g>) I can see a lot of flaws:
  • Lack of platform choice
  • Fragile single Jet database for all users of a server
  • Dependency on external directory
  • Flaws in build-in spam fighting (go ask Chris about it)
  • Complicated admin interface
  • Lack of backward compatibility and upgrade nightmares
  • Limited scalability
  • Bandwidth hog
  • No load balancing in clustering
... and so on.

Soooo what are the real advantages of Microsoft Exchange? Important: I'm not talking about Microsoft Outlook! The comparison between Microsoft Outlook and the eMail part of the Notes client is not an admissible argument for this question. Other people do the comparison there. I'm strictly looking at the server side.
From my experience Microsoft Exchange is rather accepted as collateral damage when an organization is adopting Microsoft Outlook, so that's not the explanation I'm looking for either.

17/10/2007

Lotus Notes/Domino 7.0.3 is available

Category  
The Lotus team has incorporated 1372 fixes since 7.0.2 FP2. So if you are not ready to upgrade your servers to 8.0. Did you know you can benefit from R8 servers even with R7 workloads clients?

04/10/2007

Lotus Notes 8 Launch in Bangkok

Category  
my Thai colleagues know how to throw a party. Over a thousand people showed up for the Lotus Notes 8 launch event. Having the Thai Idol finalists perform on stage might have helped a little

Notes 8 in BKK

20/08/2007

Lotus Notes and Domino --- first impressions

Category  
I can't give any first impression at this time any more having worked with R8 for almost a year from early internal builds until the final release last Friday. So let us have a look what others have to say:

Tim Tripcony: " The server installation took 4 minutes . This includes stopping the Domino service, launching the installer, navigating the installation wizard, restarting the service, and selecting "Yes" when asked if I'd like to upgrade the design of my databases."

Bruce & Julian: The taking  Notes podcast with Mary, Ed, Alan and Rob talks about Notes 8.

Vowe isn't happy about the default install, but is happy, that the old R5 desktop has been retained.

Ron Sebastian is demonstrating the new R8 features on youTube.

Nick Boldt: " Notes 8 is an absolute delight to install as an upgrade to Notes 7"

The first R8 conference is coming soon.

18/08/2007

Santosh - Composite Hero!

Category  
Santosh Kumar just published an entry on the Composite Applications Blog. He showcases how to integrate BIRT into Notes8. Finally we have some decent reporting in R8. Can't wait until that composite is available. Head over and have a look!

09/08/2007

Nathan nails it.

Category  
Nathan is right: "I've said many times that I think the great bane of Notes is that it has a history of making it TOO EASY for people to build bad applications.  Real solutions to business problems at a departmental level don't often scale well without refactoring, and I could count on one hand the number of times I've seen an organization refactor a Notes app in Notes, rather than on another platform."

11/07/2007

Domino Performance - new website

Category  
Ever wondered what makes and breaks Domino's performance? What of the gazillion settings make sense? Just head over to fastNSF and have a look. It just started and the tips are already very valuable. Check back often!


via Ed

27/06/2007

Good by W2K server!

Category  
Domino R8 runs on a lot of platforms, 32 and 64Bit alike. Windows 2000 server is not on the list. W2K server was a very popular choice for Domino boxes and a lot of system administrators were quite reluctant to upgrade to W2K3. Since mainstream support ended in 2005 IBM won't officially support a legacy platform for a new product release (It would be interesting to see if it would run). So all W2K based Domino boxes need a new home when upgrading. While W2K3 seems to be the "natural" choice, you will add the Active Directory dependency (and after all AD is "just" a fancy front-end for an Jet database - rather fragile).
Will we see a big uptick in Linux based Domino server deployments? At least the box running W2K could run Linux, giving it a new lease of life.
What are your plans?

18/05/2007

105 OpenNTF projects

Category  
Fellow blogger Thomas Adrian from Sweden has compiled a presentation of 105 projects on OpenNTF.org. He shares it on Slideshare, so anyone can embed it on a website/blog:
Like the slide deck? Slideshare allows you to download it.

07/03/2007

Designer in Eclipse --- some updates

Category  
The reactions to the DDE petition were mixed. While some were enthusiastic, others  were concerned, Maureen even cut time from her busy schedule to put things into perspective. I surly underestimated the weight people would give the fact that I'm "on the inside" (is is less then 12 month vs. 15 years as business partner), rather than seeing an initiative driven by "the godfather" of the Asean Lotus user community. The law of unintended consequences in full swing.
The development team working for Maureen was very curious to see how people would react, we had very pleasant chats when I met them at LCTY in Mumbai a few days ago. They are genuinely interested to know what features and functions Domino developers are looking for and how to create a tool that makes developers more productive. They now monitor the petition database and might some when in the future contact people how have consented to be so. We had a discussion what role XML and specifically DXL could play in a future Domino Designer. As Maureen mentioned in her blog " How can you influence the feature set? Please just ask!" So I asked and within the constrain of available time got a good conversation going. I have the impression the project is well under way and we can expect great things, just a little patience is needed (not my strength).

Now a few interesting facts:
While the global market share for Firefox is estimated around 14%, the signers of the petition seem to be more eager using it: 64.3% were on Firefox, 28.6% on IE and 7.1% on other browsers (this evaluated from the browser string which could be camouflaged in Opera or Firefox), 91.6% were users of Windows, 4.2% of Linux and 3.8% of MacOS. Only 17% of the Firefox users were still on Firefox 1.x, while only 26.4% of the Internet Explorer users had upgraded to IE7.

16/02/2007

Summer Fashion

Category  
I personally like the Lotus best.

Summer Fashion

YMMV.

Update (for Singapore only): If you want one of these, you can head over to the "Graffiti Imprint" t-shirt shop @ Novena Square #03-65/66 in Singapore (phone +65 6339 6263). Ask for the IBM t-shirts and they will show you the images and create one for you "on demand". Mugs and other stuff --- also can.

06/02/2007

Domino Chat on OpenNTF

Category  
Last century an IBM Business Partner developed a chat application that was running on Domino and was a normal NSF. When Sametime became popular this application disappeared. I always liked it for its simplicity and ease of deployment. It didn't come with a client, presence awareness, extensibility or interoperability. It just provided chat for people on the same page. One use case implemented well.
It is my great pleasure to announce, that the said business partner made that application DomChat available on OpenNTF. Go check it out!

04/02/2007

Domino Designer in Eclipse

Category  
At Lotusphere you could see something that would make your Domino Developer heart beat faster. Maureen actually demonstrated the integration of Domino Designer and Eclipse. She doesn't seem to be threatened, that Wild Bill promised to kiss her feet on release <g>. However what was absent from the 'phere was a timeline or official commitment when this will happen. I guess all Domino developers would appreciate the power of Domino Designer for Eclipse. So I created a little petition. Head over to Domino Designer for Eclipse, sign the petition and spread the word!

07/12/2006

Flowcharts from LotusScript code

Category  
Unless somebody made the extra effort to use a state-of-the-art-tool to document the LotusScript used in an application understanding the Scripts can be quite an adventure. Add a mix of JavaScript and Java and you are in for a headache. Wouldn't it be nice if we could produce flowcharts from all these. Luckily with the release of Visustin 4.0 you can document LotusScript, Java, JavaScript, C#, VB.NET, Cobol and many more. The beginning of the SendToNext Routine (from the R7 Document library) looks like this:

A sample LotusScript Flowchart
The graphics can be exported to Visio, Powerpoint, Word etc. The trial version is free.

16/11/2006

How to explain the Microsoft Architecture?

Category  
My little explanation of Expeditor has drawn a bit of attention. So let me try to do some more explaining. This would be my take on the comparable Microsoft architecture:



Let the flames come in...

15/11/2006

Volker likes Expeditor and Notes 8

Category  
Kevin had an engineer to engineer talk to Volker and now he likes the full Notes 8 client. In the comment threads there are some questions about what Expeditor actually is. At the IBM Software Universe in Mumbai I had to explain it to my audience. Here is my explanation:



Update: My English grammar sucks. Second paragraph reworded after vowe's comment.
Update 2: My take on the Microsoft architecture.

08/11/2006

eMail best practises workshop

Category  
Today I'm moderating a workshop on email best practices. I will cover writing style, management of recipients, functional subject lines etc. Over a bLaug they nicely summed it up:

Forward This Funny
To BCC or Not to BCC

03/11/2006

RSS is like the instant messaging of content

Category  
" RSS is like the instant messaging of content" is a quote in the current release of Domino Power Magazine. I like it a lot, but would rather say "RSS is to content like Instant Messaging to eMail". I guess both versions have their merits.

02/11/2006

One more reason to build Domino applications

Category  
Catching up on my blog reading I checked the http://www.joelonsoftware.com/items/2006/11/01.html on Joel on Software. Joel explains what an SQL injection bug is and how vulnerable a lot of sites are to that. An upcoming report from Mitre (cited here and here) claims that SQL injection and cross site scripting are rapidly becoming the attack vector #1 replacing buffer overflows as prevalent security hole. Joel had posted a solution how to avoid that quite a while ago. However this coding style doesn't seem to be too wide spread.
Of course you can use Domino for your web application and don't bother about SQL or if you can't live without SQL let DECS and LEI do the heavy lifting instead of "cleanscrubbing" your strings yourself.

02/10/2006

Closing down fromDomino

Category  
It has been around for a while but other than the initial stir not much activity happened. Also the team is quite busy with other stuff, so we decided to pull the plug and decommission the Blog fromDomino.com. With my team's permission I will republish the more interesting bits as SnTT articles here on this blog.

22/05/2006

IBM loves OpenNTF

Category  
Some of OpenNTF's contributors are IBMers, so it isn't surprising, that OpenNTF is well respected inside IBM. IBM used OpenNTF DominoBBS for Lotusphere and uses OpenNTF's mail experience as a source of inspiration what to do next in the mail template. But it doesn't stop there. Even in IBM's redbooks OpenNTF is officially mentioned. Have a look at page 159 in the redbook Security Considerations in Notes and Domino 7: Making Great Security Easier to Implement.
Should we have a redbook: Guide to OpenSource for Lotus Notes?

27/03/2006

What does LTPA stand for?

Category  

Is there is another meaning besides "Lightweight Third Party Authentication"?

What is your take?

stw 

24/03/2006

Who had that Ajax address dialogue?

Category  
One of my fellow Notes bloggers had some Ajax based address selection dialogue. I can't remember who. Any hint?

06/03/2006

Lotus Notes in Calgary/Canada anyone?

Category  
My colleague Nigel is migrating from Singapore to Calgary in Canada. He does Lotus Notes since R3 and is a CLP, Java and M$ certified as well. Getting certifications is a popular past time here, since it is both sponsored by the government and less boring than reading our local newspaper. He is a nice chap and if you know about a Notes/Domino related job in the Calgary area drop him an email nigel.choh (at) taoconsulting.sg.

26/02/2006

The 7.0.1 @DBLookup/@DBColumn regression error breaks Blogsphere

Category  
What an unpleasant surprise: Our admin upgraded the server to 7.0.1. Blogsphere stopped working, throwing an HTTP 500 / Invalid @Formula Text expected. It turns out, that there are quite a number of @Unique(@DBColumns) in the side block subform which now (with the regression error) fail in better filled blogs. And I have only a little more than 250 stories. Should I patch Blogsphere (categorizing the views and eventually break other things), roll back to 7.0 or wait for 7.01a/7.0.2?

16/02/2006

LAUG - Lotus Asia User Group -- 1st meeting in Singapore on 21 Feb 2006

Category  

It was very quiet around the Lotus community in Singapore the last few years. But now there will be some noise. Next week we will have our first meeting of the Lotus Asia User Group (LAUG). Our online home currently is a QuickPlace at Lotususergroup.org. Later the month we will have a Blog and other stuff for Singapore on a site of its own. The user group is designed to be regional and our friends in Kuala Lumpur will launch soon too.

Meeting details:

21 Feb 2006 - 06:45pm - 9:00pm (light refreshments will be available)

We meet at:

Raffles Campus Pte Ltd
3 Anson Road
Springleaf Tower #19-01
Singapore 079909

We have an elevator, so you don't need to climb to the 19th floor on your own.
The agenda has not been finalized yet, we will probably have a topic about Lotusphere and about the expectations regarding the user group.

If you want to join please let me know, so we can cater for you.  

08/02/2006

Lessons in Domino, Java, Eclipse and Tomcat.

Category  
Today was debugging day. DominoDAV is progressing and it looks like I can keep my promise to have the first version up this month. There are a few lessons to learn when debugging in Eclipse. I'm using Eclipse 3.1.2, Oxygen XML Editor plug-in and MyEclipseIDE as my main tools (for a complete list see my previous post). Since there are quite a number of plug-ins active, Eclipse is not exactly fast. Until you give it some breathing space. By default the memory constraint is set to 40-128 MB for the Eclipse JVM. Running a PC with 1.5G this seemed a little low. In the Eclipse directory there is the eclipse.ini file where you can adjust this setting. It is now 256-512 and Eclipse 4-6 times faster (measured totally unscientific).
When debugging nobody used System.out.println(...) anymore. Being 21st century people we use Log4J. The only drawback: It wouldn't work in Tomcat 5.5. After experimenting for a while and sniffing online I found the solution: You need to put the log4j.jar AND the commons_logging.jar (even if you never actually touch the content of the later) into the WEB-INF\lib directory. This opens the possibility to even remotely see the debug result using Chainsaw. When deploying to Websphere 6 you don't seem to need the commons_logging.jar, at least it was working here without it.
To get Notes working you need to have your Notes executable on the systems path and need to copy the jar files (Notes.jar for local access, NCSO.jar for Corba access) into the {tomcat-root}\common\libs directory. This is required, since the native libraries (a.k.a. DLLs) can be loaded only once. While there your Notes.jar can make friends with the Oracle or SQL Server jars, since they live in the same place.
Finally: When you launch Tomcat from the debugger (a nice feature of MyEclipseIDE, but also available in free plug-ins) it will ignore the path of your machine. So you are in for all the mysterious "Unsatisfied Link errors". In MyEclipseIDE there is a special setting "Library Include Path" for every server to configure. There you point to your Notes or Domino program directory and you are all set.
Finally: debugging multithread is not really fun. Since remote Notes call run in their own thread you have to take care of that. I really hope Bob is releasing (as promised considered during Lotusphere 2006) his pool manager as OpenSource soon.

20/01/2006

All my bags are packed, I'm ready to go.

Category  
Left Singapore 6am, now in Tokyo, then via New York to Orlando. 22 hours in the air, some stopover time to rest. Arrival 8pm in Orlando. See you there!

16/01/2006

Fixing a Notes database

Category  
When upgrading a server or moving datbases I like to apply a little hygiene to my notes databases. So you can find a little cmd file in the notes data directory, that fixes what needs to be fixed:

Fixnotesdb.cmd
nfixup -L -F -Y %1
ncompact -D -c -i -n -v -ZU %1
nupdall -R %1
nupdall -X %1

Running the script with the server down has the advantage, thay you can fix the address book as well. On R8.5 you want to have CREATE_R85_DATABASES=1 in the notes.ini. The only disadvantage: you need to type the password 4 times (unless your server doesn't need one). If you don't have a password, you could add:
net start "domino (lotusnotesdata)" <-- be sure to check your services for your settings!
at the end of the script and it would relaunch your server after it has run. YMMV.

Update: There is a Linux version too.
Update 2: There is a Mac version too

Update 3: for R9.0.1FP8 onwards you want to use ncompact -D -c -i -n -v -ZU -nifnsf -LargeSummary %1

11/01/2006

The contractors dilemma

Category  
I'm doing a lot of reviewing/re-engineering of Notes applications lately. I made a number of interesting observations how applications are done here in the region. Most of them were developed by developers placed into the customers environment by contracting companies or professional service firms. Once the developer is with the client they face an interesting dilemma: The client pays only for hours on the job to the contracting company. So they want their employee to stay on the job as much as possible; training gets cut short. The client doesn't consider the contract staff to be part of the firm, so no training there either. I met a number of developers who didn't get any training for 5 years, not even off days to go for their own training. While senior developers can stomach that and learn on their own, junior people will have a hard time getting the experience when they are placed into an environment where they don't have access to experienced developers The experienced ones are more expensive (nobody looks into efficiency), so quite often a bunch of newbies end up on site cut off from the experts.
As a result a lot of the code I saw was full of hacks. Hacks of someone desperately trying to figure out some something works, not hacks of someone who exactly knows what he was doing. And a lot of the code is rather fragile and won't survive an extension. Of course you could ague: a developer should take personal responsibility if not pride in the delivered results. However from the work slave's contract staff's perspective it looks different: Why should (s)he care if no one cares for him/her? On top of that: once the application is completed, (s)he moves on to the next project and so hardly face the need to maintain they system (s)he has build before, so why write maintainable code?
Here Notes and Domino are cursed. Since you can get something going very fast and Domino is rather forgiving with whatever you do, you end up with a lot of messy code. And then people say "Notes sucks". I would say: "Not training your staff sucks". I think IBM has to play some catch up to make better Notes training en vogue again.
And for code quality this is my advice: "Write your code as if the one who has to maintain it is a maniac with a gun who knows you address." (I would like to attribute that quote correctly, but I can't find the source where I did read it)

Update: I met some of my ex-colleagues from my Deutsche Bank Singapore days, who are all working for professional service firms. This little rant made the round very fast. They unisono confirmed, that they haven't seen any training for years. Luckily some of them remember the days we worked together, so they know that better quality is more fun to do.

23/12/2005

Corporate XMas eCards

Category  
eMail traffic swells during the festive season to transport all the eCards. Since it is quite difficult to distinguish between true greetings and all sorts of maleware I usually don't bother to open them. Today one arrived from a very big consultancy with the familiar " .nsf/0/" inside the URL. So I curiously had a look. Seems like they encourage their staff to use the facility instead of moving to a less trustworthy external service. Nice move, now I need to educate them how to secure such an application!

28/11/2005

To (Work)flow or not to flow --- this is the question

Category  
I'm reviewing a customer's travel request application. You know that stuff with approvals for travel, advanced cash, expense claims and a bit of travel agency involvement. My client has 5000 users using it on a 4way server and is planning to extend usage to 15000 users soon.  Users already complain, that the webUI is dreadfully slow.Currently the application is running on R5 build with Lotus Workflow 3 (the artist formerly known as Prozessware). Moving to R7 has been decided. Now the big question is: upgrade to Workflow 7 or replacing the code with custom or alternative workflow.
I think Lotus Workflow is an awesome product (Disclaimer: I'm German and Prozessware was German too). It reassembles the original concept of documents flowing through an organisation kept together by a binder quite nicely. Also you can design an awful lot of complex workflows using a simple to use workflow painter. On the other hand: our workflows are linear chains, pretty straight forward and the (web) UI is a bit too generic for my taste: Instead of approve/reject buttons you typically would have a radio button or a dropdown to set your status and then use "Activity completed" to trigger the workflow engine for the next steps. It nicely preserves a complete audit trail in the same db which keeps the indexer busy (since an agent is shuffling the trail to a different database), since we have a lot of documents created/deleted within a few minutes.
So the big question: Generic, well designed Lotus Workflow or specialized (but still configurable) speed tuned version?
Any opinions on that?

25/11/2005

Domino 7 and Websphere 6 on Windows - Part II

Category  
I finally got it working, whoha! The plug-in installation script makes 2 assumptions about the Domino installation, that were not matched by my installation: first it assumes, that the Domino program directory is the parent directory of the data directory, second, that this is not a root directory. In my case the data directory was n:\Domino7 (instead of something like r:\Lotus\Domino\Data). I mounted the N: drive under c:\data, so my Domino data directory would be c:\data\Domino7. And reran the installation. Actually I flattened out the Websphere before, just to make sure the new attempt wouldn't be haunted by zombies. This time the plug-in.xml was created as expected. However neither the DSAPI filter nor the INI variable were configured.
This time it was an easy fix. Edit the notes.ini and add:
WebSpherePluginCfg=<<path-to>>\plugin-cfg.xml
After that edit the Internet site document and add the DSAPI filter:
C:\Java\IBM\WebSphere60\Plugins\bin\domino5_http.dll
The rest of the configuration is within the plugin-cfg.xml file and according to my tests are just fine. Mental note to self: keep the Domino data directory 2 levels deep and separate program and data directory only after the final installation. On Unix/Linux of course a symbolic link would do the trick.
So it is time to develop some Webspino/Domisphere applications. Something like DominoWebDAV.
Next stop: Domino 7 and Websphere 6 on SUSE.  

23/11/2005

Domino 7 and Websphere 6 on Windows

Category  

Encouraged by the ease you can make Domino 6.5 and Websphere 6 to work together in all areas (HTTP, LDAP, Single SignOn) I tried my luck with Websphere 6 and Domino 7. Unfortunatly the Websphere plug-in installer doesn't offer Domino 7 as an option (nota bene: in R7 the location of a lot of Java stuff has moved like where to find Notes.jar). So I gave the 6.5 selection a shot. The LDAP integration worked smooth like silk, however the HTTP integration did not happen. The DSAPI filter entry wasn't created. So I peeped on the R6 installation and added the DSAPI filter name:
domino5_http.dll. When I restarted the HTTP task I was greeted with an error message:

ERROR: WebSpherePluginCfg variable not defined in notes.ini
      Define variable WebSpherePluginCfg=Location_Of_plugin-cfg.xml
      in notes.ini and restart the server

ERROR: WebSphere HTTP DSAPI filter initialization failed.  Return code = 3

Hhm. A quick check on plugin-cfg.xml files showed, that there is only on in the Plugins/config/templates directory. That for sure doesn't fit into an active configuration. Luckily there is a Plugins/bin/configurewebserver1.batcommand file which should do the trick. Unfortunatly it terminates with an error too. Well I won't let them quit to easy on me, so I had a look into the batch file. Websphere uses an amazing mix of batch files calling each other The parameters in the file look like this:

"C:\Java\WebSphere60\AppServer\profiles\default/bin/wsadmin.bat" -conntype NONE -f "C:\Java\WebSphere60\AppServer\bin\configureWebserverDefinition.jacl" webserver1 DOMINO "N:\\" "N:\\Domino7\\names.nsf" 80 MAP_ALL "C:\\Java\\WebSphere60\\Plugins" unmanaged rohandaNode01 rohanda windows

The error stated the input parameters like:
Input parameters:

  Web server name             - webserver1
  Web server type             - DOMINO
  Web server install location - N: N:\ Domino7\names.nsf
  Web server config location  - 80
  Web server port             - MAP_ALL
  Map Applications            - C:\Java\WebSphere60\Plugins
  Plugin install location     - unmanaged
  Web server node type        - unmanaged
  Web server node name        - webserver1_node
  Web server host name        - windows
  Web server operating system - ...nothing here ...

So it looked like the "N:\\" wasn't such a good parameter. I replaced it with a wild guess:   "c:\\Program Files\\Lotus\\Domino" and the installation batch file stated the right parameters and got very busy configuring the server. At this point I'm not sure if that wrong parameter is caused because of Domino7 or because the Domino server binaries are located on a different drive that the Domino data. Restarting the Domino still complained about the missing Websphereplugincfg. So back to square one and some peeping how it runs on the Domino6. There should be a plugin-cfg.xml in the C:\Java\WebSphere60\AppServer\profiles\default\config\cells\rohandaNode01Cell\nodes\rohandaNode01\servers\webserver1which isn't there.
So another try.... stay tuned

09/11/2005

Domino 6.5.4 and Websphere 6.01 on Windows

Category  
A client of mine wants to run DominoDAV. So they decided to take advantage of their Websphere entitlement for Domino. Since the entitlement doesn't state a version we decided to give Websphere 6 a shot. Having struggled with Domino Websphere integration before (Portal 4 is a beast) I told the best of my wifes I won't be back for dinner. Actually I wasn't. I was in time to pick up the kids from Kindergarten (which closes as 5:30pm). Installation runs like a piece of cake. The installer for the plug-in lists Domino 6.x in the menu and does everything. Authenticating against the Domino LDAP, LTPA single sign on worked on the first try. The instructions given in the IBM security Redbook for Websphere 5 are still valid and quite helpful, while the Domino integration part is missing from the Websphere 6 security handbook. Of course you need a big box for satisfying result. I hit only two bumps on the road, the first one due to my disbelieve it could be easy: We tried the snoop servlet myserver:9080/snoop and it worked. Then I spend 20 minutes trying to figure out under which URL the servlet would be available on the Domino port. There seemed to be no configuration so myserver/servlets/snoop didn't work. Finally I tried myserver/snoop and TADA... works as designed. So the DSAPI filter for Websphere uses the Websphere application configuration to redirect URLs from Domino, nicely done!
The second bump: When you link the Websphere to Domino's LDAP you specify the name base (like o=myorg -> /O=MyOrg ) which is added to all lookups. While this works fine for the users it seems not to work for groups (which by default are flat). I also wonder how it would work if you have more than one org in your Domain. Seems I need to brush up my LDAP skills a bit.
Now I'm itching to try that on Linux with a Domino 7.0 server.

30/10/2005

ACTION REQUIRED: Your Lotusphere 2006 Birds-of-a-Feather Abstract Has Been 'Tentatively' Accepted

Category  


Got that in my inbox this week:

Hi, Stephan and Anand:

Thank you for submitting an abstract for a Birds-of-a-Feather session at Lotusphere 2006.  We are pleased to inform you that your  BoF below has been tentatively accepted for this year's agenda.    Please advise that you will be attending Lotusphere in another capacity, as facilitating a BoF is contingent upon your paid registration, sponsorship or exhibition in the product showcase.

Please review the title, abstract, and facilitators name(s)/title carefully, and let me know immediately if you need to make any changes.  Unless we hear otherwise, this is how your BoF information will appear in printed conference materials.  Also, please watch your email for room/time confirmations.    They will be coming out in the next several weeks.

Our Birds-of-a-Feather session:

Domino as a file system --- Sourceforge DominoDAV
DominoDAV is a Sourceforge project that allows to use a Domino server as a file server using the webDAV protocol. It makes possible to read and write Notes documents and/or store files as attachments in Domino databases. A series of XSLT filters allow the processing of many XML Dialects into Domino.
We will discuss the status of our efforts, demo the achievment and gather feedback where to take the project.

Now we need to find some budget to pay for all this.  

27/10/2005

Presenting OpenNTF in Kuala Lumpur / Malaysia

Category  
The official R7 launch event was quite funny. The stage was designed like a box ring and two boxers performed a fight. The yellow fighter (labelled Domino R7) knocked out the red one (labelled "Business Pains"). The crowd of about 350 people enjoyed the difference to the usually rather stiff IBM events. Seems like hiring marketing people from Microsoft (the new kids on the IBM block in SE-Asia are all ex-Microsoft) had some impact already.
After the two round ring fight and the introduction in the R7 achievements at about 10:45 I had my turn.
I created a fast paced crossfire of slides in a mix of my own and the "Identity 2.0" style. This already had worked out well in Beijing, where I mixed big pictures, screen shots, English text and Chinese characters. (see for yourself).
About 5% of the audience had heart about OpenNTF and I saw quite some amazed faces when I introduced the OpenNTF application line-up (Bruce can you check for IP that belong to Malaysia in the OpenNTF logs?). I made them laugh quite a number of times. The slide with Singapore and the arrow was too tempting. Malaysia and Singapore have a kind of love/hate relationship since Singapore broke away in 1965. So my punch line was "I live in Singapore, in case you don't know where Singapore is, its that little renegade island down south". Of course only Malaysians laugh about that.
The local IBMers were quite surprised too and they see quite some potential for OpenNTF in the Malaysian market. Eventually they want to support OpenNTF a bit. Of course details need to be worked out. After my session in the tea break a lot of people expressed their pleasure with the presentation and asked for more details. So mission accomplished <g>.

Download the presentation overview and have a look. Full PPT on request.  

26/10/2005

IBM Lotus Notes/Domino 7 Live! --- OpenNTF

Category  

Tomorrow is the Kuala Lumpur IBM Lotus Notes/Domino 7 Live! event. While it is not on the official agenda on the web, the finalized agenda looks like this:


0900 hrs
SWG GM Kick Off by Ooi Sze Kai
Country Manager, IBM Software Group, IBM Malaysia Sdn Bhd
0910 hrs
Keynote & demo by Sean Poulley
Vice President for Business Development, WPLC Software
1010 hrs
Delivering Business Value Using IBM Lotus Notes/Domino 7 by Simon Lee
Lotus Notes and Domino Business Consultant, Asean/SA
1040 hrs
Lotus Notes and Open Source by Stephan Wissel
Principal Certified Lotus, Professional & Certified Lotus Instructor
1055 hrs
Tea-Break
1110 hrs
How Notes/Domino help in addressing your business challenges? by Ngian Siew Siong
Managing Director of Suncity
1130 hrs
By Bank Rakyat
1150 hrs
Notes / Domino DEMO by Gary Lee Sweeting
Technology Advocate for Workplace Portal & Collaboration Software, Asia Pacific
1230 hrs
Buffet LUNCH (Standing Concept)
1330 hrs
Lotus Domino 7 Server & Admn Client Update by Krista Hiltz Kahn
WPLC Business Marketing Executive for Notes/Domino Platform
1415 hrs
Application development using Lotus Domino and IBM Workplace Tools by Tan Ai Nee
Technical Consultant - Lotus Software, IBM Software Group, IBM Malaysia Sdn Bhd
1500 hrs
PM Tea-Break
1515 hrs
Supercharging your IBM Lotus Notes & Domino Investment with IBM Workplace Products and Solution by Gary Lee Sweeting
Technology Advocate for Workplace Portal & Collaboration Software, Asia Pacific
1545 hrs
Closing / Q&A by Lee Kwee Heng
Country WPLC Brand Manager
1600 hrs
END
I will talk about OpenNTF . Thanks Bruce for moral support and some of the slides.

13/10/2005

Lotus Notes 7 Launch in Kuala Lumpur - Malaysia - see you there!

Category  
The official launch for Lotus Notes 7 in Singapore is on the 20th October 2005. Unfortunately on that day I'm speaking on the 2005 Asia eGovernment conference in Beijing. I try hard, but being physically in two places so far apart at the same time doesn't work for me (yet). Luckily a week later the launch is in Kuala Lumpur, which is just a bus trip away (takes about 4.5 hours city - city vs. 3h city - city with the plane). The event will be fun, since it comes with a twist (Bruce, Vince don't squeal).

04/10/2005

NSF to PST converter

Category  
In the last 5 month I got about 1000 search referrer hits with the search topic. My short answer to that question would be: Don't do it! However as the Buddha says: "You pick the level of your suffering yourself", there is a long answer too. I can understand, that people don't know where Hannover is or can/want to wait for it. Instead of "pouring out the baby with the bath", one can stick with Domino backends and use the Outlook connector for Domino to have it both ways: Outlook on the front end and Domino as reliable, cost effective and scalable infrastructure behind. Both IBM/Lotus and Microsoft offer a flavour of it.
But if you have to go:  There are a number of 3rd party vendors who offer migration products and services: are some examples. Microsoft also provides tools and advice.
 
My personal take: Stick with Domino. Just try to get that old email backup (was it a pst?) from your Outlook/95 backup opened in Outlook 2003 or use the build in fulltext search in the Outlook client. Use the connector provided by IBM if you must have Outlook.... or wait for Hannover. upgrade to Notes 8 (You also can enjoy yourself with R7 and the OpenNTF eMail template until then).

If you want to change something for the sake of changing why not implement " Getting Things Done with Lotus Notes"?

Update:
It cuts both ways. If you came to your senses and want to move Microsoft stuff to Notes (stuff as in Outlook, Office and Sharepoint) you have plenty of help too:

26/09/2005

Sending tasks to Outlook and Notes

Category  

I'm in Munich now visiting my favourite Environment & Management software company. We are planning for the next set of features of UMsys. One feature we are discussing is the ability of the UMsys application server to send tasks/to-dos directly into the users' task lists. The UMsys server can send SMTP/MIME messages, that should be consumed by the eMail clients. UMsys' customers run Exchange/Outlook, Lotus Notes and some other mail systems. So we ran a little comparison what needs to be done for Notes and Outlook.
For Notes it is quite straight forward: you send a mime message that contains an VCALENDAR record. It can look like this:
Content-Type: text/calendar; method="REQUEST"; charset="UTF-8"

BEGIN:VCALENDAR
X-LOTUS-CHARSET:UTF-8
VERSION:2.0
PRODID:-//Lotus Development Corporation//NONSGML Notes 6.0//EN
METHOD:REQUEST
BEGIN:VTODO
DTSTART:20050930T040100Z
DUE:20051022T040100Z
DTSTAMP:20050926T195957Z
SEQUENCE:0
ATTENDEE;ROLE=CHAIR;PARTSTAT=ACCEPTED;CN="Stephan H. Wissel/TaoConsulting";
RSVP=FALSE: mailto:stephan@wissel.net
ATTENDEE;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=TRUE
: mailto:peter.pan@disney.com
CLASS:PUBLIC
DESCRIPTION:Test\n
SUMMARY:Test gMail
ORGANIZER;CN="Stephan H. Wissel/TaoConsulting": mailto:stephan@wissel.net
PRIORITY:1
UID:085D1DE1CE483B1048257088006DCAB1-Lotus_Notes_Generated
X-LOTUS-BROADCAST:FALSE
X-LOTUS-UPDATE-SEQ:1
X-LOTUS-NOTESVERSION:2
X-LOTUS-NOTICETYPE:I
X-LOTUS-CHILD_UID:085D1DE1CE483B1048257088006DCAB1
END:VTODO
END:VCALENDAR

Or one could stuff all the needed Notes fields into the MIME header, which would look like this:
X-Notes-Item: CN=Stephan H. Wissel/O=TaoConsulting;
name=OriginalFrom

When looking towards Outlook the story is quite a different beast. I did some testing with Matthias from HEC (If you need excellent Microsoft development in Germany talk to them, it's a pleasure working with people who really know their stuff well). It turns out, that Outlook doesn't understand VCalendar tasks but encodes the whole message into the body using some propriety format.
Content-Class: urn:content-classes:calendarmessage
Content-Type: multipart/mixed;

Notes doesn't understand calendarmessage thus rendering the HTML version. There I got stuck (and couldn't pull Matthias out of his meeting). Seems like it's a little harder to connect to Exchange/Outlook using open standards.

Update: We desected the MIME comming from Outlook. It seems to be formiable complicated to create a task...

21/09/2005

Welcome back Redscreen of death

Category  
I'm a happy camper in R7 land until today. I tried to drag an calendar entry (my appointment with SQ325) from one day to another. The "meeting" starts in German time and ends Singapore time and is more than 12 hours (I hope they have power sockets in coach by now). And tada, my first redscreen in R7. Didn't had any in the beta (but hadn't an SQ325 appointment then). Interesting fact: the client crashed AFTER succesfully changing the meeting. Everything looked fine afer completion of the consistency check.

17/09/2005

When Firefox and Notes become one

Category  
I've upgraded to R7 with Beta2. While 7.0 is a so called "server release" I realized the client improvements make the update worthwhile. I'm fond of the background indexing of views and the "Save window state on exit" function. The saved window state allows me to stop wherever I am when it is time to shut down (ask Anthony and Ernest about it ). I've abandoned IE for web browsing long ago and configured Firefox to launch when URLs a clicked in Notes. I would appreciate if IBM would simply use the Mozilla rendering, so the experience would be more seamless, but for the time being that's fair enough.
Both Notes and Firefox sport a similar tabbed interface, which are similar enough, so the differences get annoying. In Firefox the close button for the current window is at the end of the row, in Notes at the end of each tab. Notes saves the window state, Firefox doesn't. So me not happy (Singlish is so wonderful short ).
Enter Firefox extensions. With a little research (click on "get more extensions", click on tabbed browsing go to the last page), I found two little gems to make Firefox behave properly:
  • SessionSaver: does what it advertises
  • Tab X: puts the close button where it belongs

Now my Notes client and my browser behave similar in their tabs and I'm "happy already".

13/09/2005

R7 Replication Performance

Category  
My Domino server is hosted eight time zones away from my usual whereabouts. Replication (with compression and encrption enabled) typically yields about 20-30kbps. Until last week, then we migrated the Suse Linux box to Domino 7.0. With a R6 client I still get about the same speed, but with a R7 client it jumped up to 160kbps (which is close to the limit of my current pipe). The observation is totally unscientific and your mileage might vary. Feels like the transition from Warp to Slipstream.

09/09/2005

Ferris did it again...

Category  
Notes 7 is out for a few days only and Ferris has evaluated the impact on upgrading. They claim you can gain up to 15% efficiency mostly due to better resource utilization. Read for yourself. The original report (about reducing TCO from $22 to $6 with upgrading from Domino5 to Domino6) has moved. It looks like when they moved their website from ASP to PHP all links broke. Jakob would be furious about that.

05/09/2005

Don't put Notes Data on a mounted local drive

Category  
I added a new harddisk to my PC recently. Since all my data lived in c:\data I moved that to the new drive and used XP mounting capabilities (do I hear: they borrowed that from Unix?) to mount that drive back to c:\data.
It took me a while to assess the impact. Notes became partly unstable especially when opening mime email while replicating. I first blamed it to the fact, that I have more than one client and beta ones too. Then I realized, that in the mounted directory explorer sometimes behaved irratic, not letting me move, rename or delete files.
So I switched from mounting the drive to the 'normal' way (for Windows) and assigned a drive letter. Luckily it is just one entry in notes.ini to teach Notes its data location. Since then I had no more crashes so far.

23/08/2005

Insights into NotesSax Parser

Category  
I was working on a project that needs to import XML. Since the files could become rather big I opted not to use DXL but the NotesSAX parser class. When you are a Lotus Script die hard without much exposure to other languages the SAX parser has a "very first" for you: definition of event handlers. For those VB.NET, C#, Java and other buffs it's nothing special.
The nice thing about SAX parsers (compared to text parsing): they fix the character set for you. For example the snippet:
<company> Peter &amp; Pan</company>
 will arrive as a StartElement event, a series of Character events and one Stop event. The NotesSAX parser actually will deliver 3 Character events! Since the &amp; is an encoded entry, the text will be split into 3 and delivered as "Peter " "&" " Pan". Nota bene: there are leading and trailing spaces you can't trim. Typically you would create a steering object, that keeps the state where in XML stream you are and write out your text on a Element (either Start or End) into whatever you want to write to. I'll post some sample code for the Excel project soon.

20/08/2005

IBM blissfully absent - PACE2005

Category  
Friday and Saturday the National Infocomm Competency Centre ( NICC) was hosting PACE2005. Pace stands for "Promoting Accreditation, Competency & Employability". In short it was a conference cum exhibition about the IT industry and its jobs. All big training institutes were there, some of the higher education and quite a number of IT firms. Various awards were given out and a cute 7 year old girl was proclaimed the youngest Microsoft professional in Singapore.
The IT firms participating were: Sun, Microsoft, Red Hat, Novell, Oracle, CheckPoint, BusinessObjects and Iomega.
IBM was blissfully absent. Maybe I'm getting it wrong, and caring for the next generation of passionate users isn't important, however I was quite disappointed. There is so much cool IBM technology around but IBM (at least here) fails to create excitement among the IT professionals of tomorrow.
I talked to some of my training friends, they said demand for Lotus training is non existent, however my recruiter friends see an increasing demand for Lotus know how, which leads to a painful gap, that Microsoft is exploiting: "See you can't get Lotus people, why not migrate to our offerings" -- and they are right (at least here).
Where is that "Singapore's coolest RCP application" contest?

16/08/2005

Importing a MS-Excel XML Spreadsheet to Domino

Category  
Head over to fromDomino and read my latest post about cleaning up a MS-Excel XML Spreadsheet to get ready for an import into a Lotus Domino database. This post covers the XSLT. The LotusScript including the sample database will follow later.  

05/07/2005

I need a DXL/XML guru's advise!

Category  
I'm working on a project, that will reunite a bunch of databases that once were one back into a common template. One tool that worked very well is DXL export, XSLT transformation and importing that stuff back. I'm using the Body field of a Notes document to store the DXL, which creates some kind of documentation/audit trail. This morning we hit the wall.
I exported a rather complex form (a lot of subforms, multi-nested tables, hotspots and a lot of fields). When I tried to reimport the DXL untouched I was greeted with this error dialogue:

(For RSS readers: the message says: "Left margin cannot be greater than right margi". I did the msgbox, but that was the error in the DXL importer. Please note the typo/truncation. Anybody came across this and could advise what to do?

Update: I found a solution (and opened another can of worms)! The form I tried to export has quite a bit of "history" So in the table tag there is an attribute r4spacing="true". With this attribute set I get the error message. The attribute tells the designer to use the fixed values provided in refwidth="10.4778in". Once you remove the attribute "r4spacing" import works. Of course now the table rendering follows a different rule set, so it gets all messed up. Looks like another session in XSLT to clean this up.

This is the agent we use for import:

31/05/2005

Another of Domino's magic numbers : 42220

Category  
While cleaning up my mail file (hovering at 2 GB) I dropped my full text index on the server. I forgot to recreate it. When checking the log files I discovered one of Domino's magic numbers:

31.05.2005 12:17:40   Warning: Agent is performing full text operations on database 'test.nsf' which is not full text indexed.  This is extremely inefficient.
31.05.2005 12:17:40   Full Text message:  index of 42220 documents exceeds limit (5000), aborting: Maximum allowable documents exceeded for a temporary full text index
31.05.2005 12:18:25   AMgr: Error executing agent 'Move Document Around' in 'test.nsf': Maximum allowable documents exceeded for a temporary full text index

So you can't have more than 42220 documents if you want to full text search an database without a fulltext index. But who would want to do such a thing anyhow. Harddisk space is growing faster than CPU or transmission speed.

31/05/2005

Nupgrade 3 - Lesson learned

Category  
I'm converting a customer to Lotus Notes. When transfering the local Outlook data I hit the wall with one machine. It only would import data from an old PST file but not the main file.
Investigating a little bit I found out, that the old file once was the primary local store and had also a "Personal Folders" folder and structure.
Once this file was removed the nupgrade process would run nicely converting all the 1.5 GB into Lotus Notes. So lesson learned: make sure there is only one "Personal Folders". I wonder if it is the name or some internal id nupgrade is using. Another can of worm has been opened and I'm stuck there: I tried to create a new Outlook profile with Lotus Notes as provider. It recognizes the DLL (nwnsp32.dll) correctly but terminates with (funny) error messages: When I try to create a profile with Lotus Notes: "A service provider could not be loaded. Contact your administrator"; when I try to add Lotus Notes Mail as a service and click on properties: "The action could not be completed. The action could not be completed." (Yes twice!).
Googeling for the problem makes me shiver, since it seems one of the hard to track down multi vendor semi standard mapi issues.
Any takers?

11/05/2005

Domino on Solaris anyone?

Category  
I encountered a number of installations here where Domino is running on Solaris on SPARC I found at least one reference of Domino running on Solaris 10. However I didn't find information about running Domino on Solaris on x86 platforms. I'm particularity interested in the combination Solaris10/Domino 6.5.x. The official IBM site is a bit blur on that. Sun has these neat AMD based servers, so I wonder:
a) does it work?
b) how does it perform?
Anyone enlighten me?
stw

Update:
On IBM's official platform page for Domino Solaris 10 is not listed. Solaris 8/9 are listed without a platform mentioned. At least one engineer has confirmed, that Domino7 will be certified on Solaris 10/Sparc. Finally - well hidden in the release notes of Notes 5.0.11 there is the confirmation, that Domino6 will not run on Solaris/x86. They could state that on the platform page.

Update II (July 2005):
Christophe tried it and failed. However there is a new hope. Lxrun might make it possible or even more likely: since IBM renewed their contracts with Sun, a future release might have Solaris x86 on the roadmap.

26/11/2004

Old challenges resurface --- The Domino upload control is still a mystery --- can you solve it?

Category  
When I was posting about my XML wish list, I remembered that I had that wished before. In 2001, when Gary Devendorf was in charge of XML in Domino I already suggested the enhancements. You can see links on the R7 beta forum.
When going through the archive I also found, that I tried to demystify the Domino upload control. It first became available in Quickplace and made it's debut on Lotus Domino in version 5.08 with the first incarnation of Domino Web Access (a.k.a iNotes). I tried to figure out how that control worked, since I wanted that in my own application.

25/11/2004

My X-M(L)as wishlist for Domino 7

Category  
I posted the link to my post @ IBM already on fromDomino.com. I was wondering how many of you:
a) like the suggestions
b) would like to see them implemented ( Bruce Elgort does)
c) will help to promote the idea
And what would it take IBM to actually put it on the ND7 feature list.
stw

12/11/2004

Domino 6.5.3 - DominoCompleteDoctype - Be careful what you wish for.

Category  

When I did read about Domino's fix for the incomplete doctype on the:gutted:geek I was itching to test it. After upgrading the server to 6.5.3 I activated DominoCompleteDoctype=1 (you can use 1 for HTML 401 transitional and 2 for HTML 401 strict) only to see most of my web databases fail: CSS rendering was gone and JavaScript didn't load. Among the applications fail also was Blogsphere, the template this Blog is build on.
We had a public holiday yesterday (the Hindu Depavali, it has clear advantages to live in a multi cultural environment, Monday Hari Raja holiday follows), so I got some breathing space to investigate. It turned out it was a case of old R5 work around and subsequential "Chin Chai" work (sloppiness). Before R6 we used to put JavaScript and CSS in pages and forms. And since you can twist and tweak JS/CSS with @formula in a form/page I often keep them there (one of my favourite tricks: generate JavaScript out of the document I am viewing using a special view and a form formula, makes design much less cluttered). Domino delivers them as mimetype text/html.
As long as browsers run in Quirks mode it doesn't matter. The very moment you force them into standard mode all of them get picky about the mime types. So you need to change the mime type in your forms/pages that contain only css/js to "text/css" or "text/javascript". Luckily Domino allows you to do so on the form/page properties.
There is another catch: In standard mode CSS now seems to be case sensitive. class="rightcolumn" would not match .RightColumn { } anymore. (Now I would wish for a tool that shows case mismatches in CSS vs. HTML). You need to fix you html or css.
So happy testing of your web applications!  

09/09/2004

Domino Admin 6.5, Dirlinks and a bug

Category  
My domain in Domino development, so I never paid much attention to the Domino Administrator client. So from time to time I have to learn a new trick there. When looking at the files tab you usually see a tree on the left and the files on the right. There is a dropdown where you can select the type of files you want to see: . What if found out (OK I never bothered to read the manual), is that you actually can click on the "All" button right of the dropdown and switch to a flat view of all files (I always thought this is a filter function and never tried it before). The button then changes to this: [IMAGE] so you can switch back to the tree style display. This comes in very handy when you want to operate on all files.

I discovered one serious bug there (maybe it is already documented, so the discovery would be for myself only): In tree view subfolders of directory links do not show! We use a lot of directory links (one for each customer), so it's kind of weird. So files that are inside subdirectories of linked directories do not show in the Admin either.... unless you switch to All view, then they are neatly listed.

17/05/2004

Cleanup your Domino Directory Group structure

Category  
When you run a large Domino Installation with a high fluctuation of people coming in and out, you might end up with a lot of Zombies in your groups. This might happen especially with part time administrators, who are to busy with their main jobs (and don't use AdminP to delete users). I had to do a clean-up exercise recently and I used the simplicity of @Formulas to remove user names that are no longer in the Domino Directory from all group entries where they might still be in:
Step 1: Create a new view named (GroupCleanup)|GroupCleanup with one column. Sort that column.
View selection formula: SELECT ( Type= "Person":"Group":"Server" )
Column formula: @If(Type = "Person"; @Subset(Fullname; 1);Type = "Server"; @Subset(Servername; 1); @Subset(ListName; 1))
Step 2:Create an agent, type formula to run against selected documents:
SELECT @All;
v := "GroupCleanup";
c := @Elements(Members);
good := "";
found := "";
@For(i:=1;i<c+1;i:=i+1;
current := Members[i];
found := @Trim(@DbLookup("Notes":"NoCache";"";v;current;1;[FailSilent]));
@If(@Contains(current;"*");good :=good:current;good :=good:found));
finalgood := @Trim(@Unique(good));
finalbad :=@Trim(@Replace(Members;finalgood;""));
FIELD removedMembers := @Trim(@Unique(removedMembers:finalbad));
FIELD Members := finalgood;

Step 3: Use an agent to remove groups that don't have members anymore. Run the agents multiple times to get rid of nested redundant groups:
SELECT type="Group" & @Trim(members)="";
@DeleteDocument;

Caveat: This will remove all user names that are not in the main addressbook. That means all users of external organizations might get lost too. You could extend the formula by adding more @dblookups to additional addressbooks.

Disclaimer

This site is in no way affiliated, endorsed, sanctioned, supported, nor enlightened by Lotus Software nor IBM Corporation. I may be an employee, but the opinions, theories, facts, etc. presented here are my own and are in now way given in any official capacity. In short, these are my words and this is my site, not IBM's - and don't even begin to think otherwise. (Disclaimer shamelessly plugged from Rocky Oliver)
© 2003 - 2017 Stephan H. Wissel - some rights reserved as listed here: Creative Commons License
Unless otherwise labeled by its originating author, the content found on this site is made available under the terms of an Attribution/NonCommercial/ShareAlike Creative Commons License, with the exception that no rights are granted -- since they are not mine to grant -- in any logo, graphic design, trademarks or trade names of any type. Code samples and code downloads on this site are, unless otherwise labeled, made available under an Apache 2.0 license. Other license models are available on written request and written confirmation.