Data integration then
Notes 3 introduced the ability to do @DbLookup and @DbColumn using ODBC data sources. Lotus then bought NotesPump, a data integration engine that let you map data fields between an external source and Notes. NotesPump provided a set of "connectors" that let a LotusScript developer work with external data using the same API, regardless of the back-end data source structure.

IBM released other integration tools as well, most notably LotusScript Data Objects (LS:DO), a set of drivers that "wrap" ODBC sources to make it easier to work with them from LotusScript. With Notes/Domino 4.5, IBM integrated the "real-time" capabilities of NotesPump into the Domino server, and rebranded it DECS. This let you quickly provide real-time access to external data sources in your Notes applications simply by filling out a few configuration forms. But, there were some drawbacks; most notably, for every external record you brought into Notes, you had to have a corresponding "key" document. This defeated the goal of using Notes as a pure "front-end" to external data because you still had all those documents in your Domino database.

IBM took the part of NotesPump that was left after adding DECS to Domino and rebranded it Lotus Enterprise Integrator (LEI). It provided scheduled integration between Domino and external sources and gave you the ability to have additional activities, such as scripted agents you could use to massage data moved between the two data sources. LEI included a new set of data drivers known as the Lotus Connectors. DECS had Lotus Connectors, but the connectors provided with Domino and DECS weren't the same as the connectors included with LEI. Things were getting better, but we still didn't have an easy way to integrate external data with Domino.

Integration now
Jump forward to today. One of the primary goals of Notes/Domino 6 is to make integration with external data as easy as possible. This is accomplished a couple of ways. In previous releases, DECS required you to configure the connections through the use of a database called the DECS Administrator. The person who was the Domino administrator usually controlled this database, so not only did you have to create the databases that used DECS, you also had to coordinate with someone else to configure and enable the connections and activities in the DECS Administrator.

Notes/Domino 6 puts the power back in the hands of the developer. Domino 6 includes a new design element called Domino Connection Resource (DCR) that represents the connection document found in the DECS Administrator. After you have a DCR, you can indicate in your forms that the data provided for individual fields comes from an external data source, which maps to your DCR. But IBM didn't stop there.

Getting virtual
The exciting new Lotus Enterprise Integrator, LEI 6, introduces a concept called "virtualization," which lets you have a Domino database act as a pure front-end to relational data, with no documents in the database. That's right -- views, forms, LotusScript, formulas, etc. all work as if the database has documents, but there aren't any. These are virtual documents. Additionally, you can have real-time virtual fields (the same thing DECS does) that map to virtual documents, providing a powerful and flexible model for front-ending data. LEI can also present multi-valued fields on a document, where the multi-value fields are multiple records in an external data source. You can even virtualize attachments and rich text fields.

LEI and DECS now share the same Lotus Connectors, so if you use them in your LotusScript applications, the connectors you have before you install LEI are the same you use after installation. The connectors are also more scalable and fast.

With LEI, you can provide the power of Notes with the scalability of an external data source -- and isn't that what we've wanted all along?

Looking ahead
And here's something to watch: IBM Senior Vice President and Group Executive Steve Mills has said that a future goal is to offer an alternative relational back-end data structure for Domino based on DB2. IBM tends to test ideas in other applications, to learn about the functionality in the real-world. An example of this is Domino Off-Line Services (DOLS), which began as the offline feature of QuickPlace. After testing and validation, IBM packaged it on its own and then made it a part of core Domino. I believe virtual documents may be another such technology trial, proving it's possible to provide a relational back end to a Notes/Domino data source. So, if you want to see where Lotus Software is going, look at where it is now. Based on what I'm seeing now, the future of data integration in Notes/Domino looks bright.