Transactions (Editorial)

Many are experiencing the problems with NVFS on the new Tungsten T5, E2 and Treo 650. The problem lies in how PalmOS handles databases and how it was not ready for such a back store design. When PalmOS was first written, they chose a combination of ROM (Read-Only-Memory) and RAM (Random Access Memory) to provide storage and dynamic memory in one. They used ROM to store the data which you would find in your device after you bought it more like factory settings. RAM on the other hand was split into two parts: Dynamic memory and Storage where dynamic memory was read/write directly while reading storage was just like dynamic memory but writing had to be done with a special DmWrite function. The advantage was plain obvious. Instead of having a file system, PalmOS just has memory chunks chained to each other which form the databases and records we all use. However there was a problem with this design. While it was wonderful in performance terms it was too direct. If I wanted to lock a record, I would get a memory address which I could ready with direct manipulation and if I wanted to write to the record, I just pass the memory address to DmWrite. If the record memory address moved (when unlocked), I would have to get its address again by finding it through its database-record relationship. So where is the problem? First of all... Each internal card slot (which will go away with Cobalt) has a maximum of 256 MB. While this was a far away limit with the first Palm Pilot it sounds alot like Bill Gates when he said that 640K was enough for anyone. Limitations in a design show in the end in one way or another and PalmOS hit that limit. We now have devices with 128MB and they knew that they had to solve this sooner or later. So how could they solve it? Well... along came NVFS which is a Non Volatile File System which is basicly like an internal SD Card. But unlike RAM, you cannot read/write directly, you have to do buffer based reads and writes much like you do with VFS. So how to make current applications compatible? So they basicly made it work like a desktop system (and PocketPC) where you have real RAM which is like your active working memory and a seperate storage which like windows has FAT/NTFS, PalmOS has NVFS. When ever a programmer opens a database, it copies the database and all its records to a temporary location in RAM and writes it back when you close a database. The problem was that many programmers took advantage of the direct read/write and were reading/writing to records even when thier databases were closed. The second problem was that the amount of RAM they used as the temporary memory for databases was only about 10MB which means if an application opened two many big databases the application would burn and crash. The temporary amount of RAM will grow and grow just like the desktop systems do but then you will get the famous saying 'there is never enough RAM'. What is needed is for developers to move out of the direct read/write design and move into a transaction design. A transaction is a group of actions which are executed at once. So instead of reading and writing directly where if the application crashed in between writes you would have an inconsistent database or record state, you would specify all the operations and then do all the write operation in a single push. What are the advantages of transactions? 1. Because writing is done in a single action, you have more centralized points of failure and it allows your application to be either working or not working without the gray area of inconsistent or invalid data. 2. Because reading and writing is not done with direct memory addresses, it allows the amount of store to grow without having to increase the temporary memory because nothing is directly accessed rather you call a function to read some data into a buffer (which is stored in the temporary memory but is much smaller than the entire database or a group of records) and another function to write data back to the store. 3. We will able to take advantages of mini hard drives or remote store like an FTP site. So instead of being limited by MB's, you have Gigabytes of memory to your use. So what are the disadvantages? Speed Buffer based read/write are always slower than direct memory access. However depending how you handle your memory its not that slower. I dont have benchmarks but I believe that speed is less of an issue when it comes to reliability, expandability, flexability and so on. I believe that if PalmOS goes more towards this direction, they will save themselves many problems in the growing storage direction and more reliable need of the system.