Update - The train is moving

I finished coding the following components which means nothing to most: CoreObject, CoreMember, CoreFolder, CoreProperty, CoreRectangle, CorePoint and CoreNumber Next week I hope to finish: CoreDate, CoreTime, CoreString, UILayer, UILine, UIRectangle, UIPixel and UIColor While this does not mean much to programmers and users let me explain what this means when I finish those components next week. I will be able to put the code in some prc and give it out you all so you can see that I am definently working on something. I'll probably do some kind of animation with the layers so any simple animation ideas are welcome (that use line, rectangle, pixel combinations). The Layering engine uses one single off-screen buffer where it paints the contents of the layer clipped to the buffer and then copies the buffer to the display. Note that it does not refresh the entire display each time, rather only a clipped area which needs to be updated (called the update region). Becauses it paints layers contents one on top of each other you get natural transparency without using transparency colors. Any area not used by a layer is transparent.

Update - Change of direction

Because of a few events that happened this week I will be putting the multi-threading on the side (now that I know it works) and I will be working on the object storage itself. I think its time to actually get this object storage out the door. Which is why I am going to code like furry this week and next week to finally get this bit ouf of my way because while it has an important aspect, its taking too long.

It Works!

OK... The multi-tasking works where it can switch between two threads concurrently. I now will take the code and encapsulate it into the CoreThread component which I will code this week. This means that Screens will have threading capabilities which is great! The kernel will use this threading capabilities to start the module calls so that modules can run concurrently. This is done by a module saving its state (registers and stack) when it yields (which a kernel API can flag the module to yield) and then closing itself. When the kernel re-runs the same module it passes its state back so that the module can restore it. Because the saving and restoring is done by the module, its not interface dependent or even language dependent. I am so happy with how the project is moving in the last few months in both code & design. Note that each thread has its own stack of module calls where each call in the stack has its own state. The current implementation uses ErrSetJump and ErrLongJump to save/restore the registers and copies the stack between an initialized variable at the beggining of the stack and a variable at the end of the stack. It will be currently stored in feature memory. While this is slower than setting/getting the A7 register, its more implementation independent and wont break on cobalt which many other implementations do.

Transactions (Editorial)

Many are experiencing the problems with NVFS on the new Tungsten T5, E2 and Treo 650. The problem lies in how PalmOS handles databases and how it was not ready for such a back store design. When PalmOS was first written, they chose a combination of ROM (Read-Only-Memory) and RAM (Random Access Memory) to provide storage and dynamic memory in one. They used ROM to store the data which you would find in your device after you bought it more like factory settings. RAM on the other hand was split into two parts: Dynamic memory and Storage where dynamic memory was read/write directly while reading storage was just like dynamic memory but writing had to be done with a special DmWrite function. The advantage was plain obvious. Instead of having a file system, PalmOS just has memory chunks chained to each other which form the databases and records we all use. However there was a problem with this design. While it was wonderful in performance terms it was too direct. If I wanted to lock a record, I would get a memory address which I could ready with direct manipulation and if I wanted to write to the record, I just pass the memory address to DmWrite. If the record memory address moved (when unlocked), I would have to get its address again by finding it through its database-record relationship. So where is the problem? First of all... Each internal card slot (which will go away with Cobalt) has a maximum of 256 MB. While this was a far away limit with the first Palm Pilot it sounds alot like Bill Gates when he said that 640K was enough for anyone. Limitations in a design show in the end in one way or another and PalmOS hit that limit. We now have devices with 128MB and they knew that they had to solve this sooner or later. So how could they solve it? Well... along came NVFS which is a Non Volatile File System which is basicly like an internal SD Card. But unlike RAM, you cannot read/write directly, you have to do buffer based reads and writes much like you do with VFS. So how to make current applications compatible? So they basicly made it work like a desktop system (and PocketPC) where you have real RAM which is like your active working memory and a seperate storage which like windows has FAT/NTFS, PalmOS has NVFS. When ever a programmer opens a database, it copies the database and all its records to a temporary location in RAM and writes it back when you close a database. The problem was that many programmers took advantage of the direct read/write and were reading/writing to records even when thier databases were closed. The second problem was that the amount of RAM they used as the temporary memory for databases was only about 10MB which means if an application opened two many big databases the application would burn and crash. The temporary amount of RAM will grow and grow just like the desktop systems do but then you will get the famous saying 'there is never enough RAM'. What is needed is for developers to move out of the direct read/write design and move into a transaction design. A transaction is a group of actions which are executed at once. So instead of reading and writing directly where if the application crashed in between writes you would have an inconsistent database or record state, you would specify all the operations and then do all the write operation in a single push. What are the advantages of transactions? 1. Because writing is done in a single action, you have more centralized points of failure and it allows your application to be either working or not working without the gray area of inconsistent or invalid data. 2. Because reading and writing is not done with direct memory addresses, it allows the amount of store to grow without having to increase the temporary memory because nothing is directly accessed rather you call a function to read some data into a buffer (which is stored in the temporary memory but is much smaller than the entire database or a group of records) and another function to write data back to the store. 3. We will able to take advantages of mini hard drives or remote store like an FTP site. So instead of being limited by MB's, you have Gigabytes of memory to your use. So what are the disadvantages? Speed Buffer based read/write are always slower than direct memory access. However depending how you handle your memory its not that slower. I dont have benchmarks but I believe that speed is less of an issue when it comes to reliability, expandability, flexability and so on. I believe that if PalmOS goes more towards this direction, they will save themselves many problems in the growing storage direction and more reliable need of the system.

Damn... no luck

OK... so its not going well... but I am optimistic... I found some article on cooperative multitasking which is nearly portable so I will be coding/testing it out next week. This is realy important feature and I am working basicly day & night to get this done as soon as possible, I dont want to take another few months just to solve this. So lets put this aside for a minute and talk about the UI: So what do have in plan for Screens? Its nice that I expose all these programming stuff but what realy interests users is when will it come out and what will it have/do. The first thing I have been terrible at predicting so I just stopped doing so (It was meant to be released 3 years ago). However I can shed some light on what I at least plan to be usefull in the UI which I am always refining. First of all, Screens is going to be very search oriented so for example alot of functionality is built in the basic controls. All list, tables and tree controls are searchable, sortable and filterable. For example if you start typing letters while a list, table or tree is active, it filters out the entries in the list/tree/table to only show matching results. For a tree, it expands the items but still exposes thier node structure. So you can know not only the result but how to get to it. This makes it very usefull when dealing with structured information since it can stay structured and you can search through it. For example when you want to open a document, it shows you a browser dialog however instead of having to expand the tree items to find the document you want, just type the name of the document (or part of the name) or any other meta-data which matches the document and the tree will filter out any non-matching entries so you can find your document much faster. This is going to be an addictive feature because its not obstructive and still keeps structure instead of hiding it like other methods (queries). I'll post some more of my UI ideas soon

Update

OK... So I have trying to debug the multi-tasking code yesterday and today and no luck. I just cannot call MemMove from and to the stack while its in use. I will use a different technique which is the A7 register stack pointer. Hopefully this will work since so many developers tell about it. People have no idea how hard these things are to design. The proccess should be quite simple where I copy the stack to dynamic memory and set the A7 to the copied stack. When I want to restore the stack, I just set the A7 back to the original location and Jump to the right location in code. This way I always have the threads stack in a saveable location in dynamic memory. Cobalt doesn't allow to pass dynamic memory through SysAppLaunchRemote calls so I use dynamic memory to pass the stack but use feature memory to pass it back which means that on Cobalt it will be slower although the proccessor should be faster anyway so no big deal hopefully. Garnet on the other hand just passes a system dynamic allocated stack which is used and switched between application launches. Once I know that I can swatch threads locally, I will write PublicThread component which will allow the object storage to be able to multi-task (so you can copy an object and switch to another thread in between).

Update

OK... Most of the multi-tasking test works, I am able to switch between threads and although I was stuck on a bug when restoring a thread (I was using a local variable when restoring the stack of a thread), I think I know how to solve it so hopefully the stack restore should have no other bugs. Note that variables are intact after copying the stack but I did have a nasty bug that it was killing the code when the current function ended, but hopefully I solved that so it should not happen. So... If I succeed and that thread switching works... what happens next? Well first of all it means that I will be able to implement the semi pre-emptive multi-tasking for Screens which means that programmers can code as if they were the only application and do not have to specify yield points since any API function can yield if it wants to. This means unlike cooperative multi-tasking where you need to call a yield function to yield, Screens does this for you however you can still freeze the system with a simple endless loop. But dont worry... UI is seperate from Code (unlike Windows where a window is attached to a thread) which means that even if the system restarts, your current working data stays intact. So once I know that the thread switching works fine then I start coding the components such as CoreThread, CoreStack and CoreMessage next week. So what does this help me as a user? So all this techno-babel, but it does help the user that I have invested in this design. When you copy a file from your device to your handheld, does your music hickup? Does your device freeze so you cannot do anything untill the file finishes copying. Would you like to download files while browsing or look at your emails while new ones are comin in? If you do... Thats what this design helps... It gives Screens the power to handle all these tasks. When you drag & drop an object lets say a database (prc/pdb) file into a VFS Folder, instead of showing you a progress dialog with a cancel button in which you either wait for the file to copy or cancel the operation, Screens shows the progress in the actual copied icon. It shows the icon in the VFS Folder a bit shadowy to show that it does not exist yet and shows under the label the percentage done copying. You can either tap on the icon to show the progress in a dialog or do anything else. This is not the co-routines multi-tasking that others are using. This actualy can restore a threads state and stack so you dont have to code to make your subroutines small. Also because I deal with semaphores for you, you dont need to worry about critical sections since all objects file stream have a request queue so while many threads can read an object only one thread can read & write to an object. If you create an object and lock it, anyone trying to read or write to the objects members are also blocked untill you unlock it. This is much easier then dealing with semaphore and mutex combinations.

Update

Well... it looks like my prediction was wrong and I am on the computer just like any other week. Oh well, just hope that my house will be clean by the end of the week before passover. I know that the posts are very technical which is basicly because thats what I am dealing with at the moment. After I finish the kernel, I promise to post more user friendly information.

Update

So... The object storage so far is doing well but I am a little worried about the classes design wether it should just be lookup or merging. I'll check out to see what is better and see how the design handles it. One of the biggest problems is recursing which means what if object A is derived from object B which is derived from object A? This kind of situation should not happen so I will work on how it wont happen. Also I need to deal with sub properties/methods of a class inheritence, so hopefully I will find a solution by the end of the week. I know that most of you probably haven't got a single idea what I am talking about... but oh well... The multi-tasking design is also going well and I should try to code it throughout passover where I will see wether the stack can be restored or not. In theory it should work and any other OS has done this so it should work although I am bothered a bit by the performance hit but I can always optimize it later. If it works and is fast enough, then I will start working on the object storage code. This is great progress so far and once the multi-tasking and object storage is coded I have very little left to code to finish the kernel. The multi-tasking design was the hard part and since I found how to solve the communication model between modules, Its either a make or break situation. The Core API will be available in the kernel release along with the kernel source code. I think it will be released in GPL but if anyone has any other licences I should check out, let me know. The Core API is C/C++ wrapper based but can easily be made to support other languages as long as they support stack manipulation (save/restore) and PalmOS API access. It will have documentation and I plan to launch a web-site when I have the kernel ready for release where you can download the code, source and documentation. Remember that the kernel is not a library, you just make sure that the Screens.prc is on the simulator/device and just compile your code with the wrapper header and source files. No linking to anything, just compile and launch as a normal application. It registers the module with the kernel and then gets put in the thread loop automaticaly. Once you try it, you will see how easy it is.

Update

Hi Everyone, This week is preperation for Passover so I wont be on the computer hardly at all however I am still be thinking away... I have doing alot of thought on the multi-tasking design and so far it sounds very feasable although slow. I talked about it with my father and he liked it. Why is it so slow? Well because of the way the stack is stored/restored in each message switch. Because of limitations I had with making the system compatible with Cobalt, I had to use Dynamic memory and feature memory to deal with passing and returning memory. This means that the stack needs to be copied from an object to dynamic memory and then again to the local stack which is quite a performance hit. When does a switch happen? On any API call which involves calling another module execution. So if you call UIWindowDraw it is actually switching from your message to another message which deals with the window drawing. This will be tested to see how slow it is but in general, there is quite a bit of general traffic slowdown in theory. If it gets to slow, I might allow the API to be able to execute sub-calls directly without having to switch but then you get an even more slow-down when having to switch to another thread, so I will see how it works out. So what else has been happening? The object storage has been through a few changes. One of the problems with a reference only object storage was that you could get chains of orphaned objects where they referenced each other but you had no way to get to them from the root object. To solve this, all objects are stored in a tree just like a normal file system however each object is a file and a folder in one. Every object can also hold references to any other objects. However references and child objects have no difference apart from a single function CoreObjectIsLink which can check if an object is a reference or a child under a specific object parent. Another thing is links (references) do not have a handle, you cannot point to them because technically there have no representation. If you try to get the handle of a link, you get the handle of the object it points to. If you delete an object, it deletes all its children and all thier references. However if another object references any of those objects, the references are removed when they are encountered. This saves the need of searching the system for references. However each object has a reference count so that it can know how many objects are referencing it. This is a bit like WinFS and allows an object to act on its reference count to do things like removing itself if it has no references. This is one of the reasons that the object storage has to be multi-tasking because if it is going to execute methods on objects it needs to be able to switch to other threadswhile doing maintence work. Because when you delete an object, it first notifies all child objects of this operation and if they are deletable, then it goes through all child objects removing them and only then removes the parent object. If any child object fails to be removed (because they did not handle the previous notification) then the parent object is notified of this failed remove and it can decide what to do. Note that static objects (that dont have a new/free method) dont get notified (because they have no constructor/deconstructer).

Messages

Screens has the concept of modules, threads, messages and methods. Modules are the code. Each module is a prc application with a normal PilotMain routine but which handles the Screens Launch code. A module does not have globals or static variables or even muliple code segments. This is due to the PalmOS design for speed purposes. Each module has a main thread which runs. A thread is a stack of messages where messages run in a first in, last out design. Each message corrosponds to a module handling some method. If two threads exist they can run concurrently by switching thier active message into the kernel for CPU use. This means that even if one thread is blocked, other threads can continue to proccess thier message stack. Each message has its own local variable stack which is stored/restored on every message switch. Because the of the mecanizm that messages are run, on PalmOS Cobalt they run protected so that one message cannot affect the memory space of another message. Each method is executed via a message. A method has a unique name which is 128bits (32bit CompanyID, 32bit ModuleID, 32bit ComponentID, 32bit Name) long and because its totally unique there is no conflict no conflict with other method mecanizms like Strings. Each method has a data block which its called with and and another data block it returns to the caller. It might sound a bit complex but most of it is dealt with for the developer. This includes the biggest pain of multi-tasking and multi-threading which is semaphores and mutexes. For example in a single tasking world (like PalmOS apps, not the PalmOS kernel) you dont need to worry about the same resource being modified while you are reading it. In a multi-tasking world you need to worry about shared resources by locking them and unlocking them when you dont need them. Screens automaticly handles requests to read/write calls to modify an objects stream, so a developer just calls CoreFileOpen with either the read-only or read & write flag and Screens will automaticly determine if the request should wait or continue. If the request is to wait, it pauses the thread and continues it when the file is ready. It allows multiple read requests but only one exclusive read+write access. But developers can just open and close files as normal and Screens handles this for them. It is done for all objects which is the only shared resource modules can pass to each other. I will provide alot of documentation when I release the Screens kernel, so that it will be very easy to understand.

Multi-tasking design

As you all know, Screens wants to bring semi pre-emptive multi-tasking to the PalmOS platform. Why Semi? Because its cooperative internally but it switches not on a single yield function but can yield on any Screens API call. So as long as the developer does not do an endless loop without calling an API call, the system will continue running. So how will I do it? I have researched this subject for a long time and this is my theory: I use setjmp() at the start position (like just after the module gets called via its PilotMain function) and store it somewhere. The code then runs as normal. When I want to stop the current module thread to switch to another, I save the end position (still using setjmp) and its stack (from the start position to the end position) and then use longjmp() to jump to the start position. This then of course exits the module returing to the scheduler. When I want to restore the modules execution, I run the module again and it saves its new start position but then copies the stack back and jumps to the end position and continues execution from there. Why theory? Because this all works apart from the restore operation which might have unexpected results. So that is what I am going to test. If it works, I will have multi-tasking in Screens so you wont have to code in a single-tasking environment anymore. While there are limitations in my design such as no globals (but thats because modules cannot have globals, instead use the object storage) and only local variables are stored (and the required registers stored by setjmp) so you cannot use anything outside that world. But nothing that normal developers use anyway (dont tell me about signals... most apps dont touch them). I am going to spend the next month to try and see if it is possible. Where will I store the stack? Not in an object but rather in feature memory. Why not in an object? Well... in the end it is stored in an object but the actual switching code cannot make any Screens API calls because the stack does not always continue after a SysAppLaunch call, so I cannot be sure that I am saving the right stack. The design I have just told you should work on any PalmOS version including Cobalt. Wish me luck! Because if this works... Screens will have a very compelling reason to code for it.

Update - Coded CoreStream and CoreFolder

I finished coding CoreStream (renamed from CoreFile) and CoreFolder. You might have thought how I can I code these components so fast. Well as I told you before all these components are basicly building on the previous components functions. So for example CoreObject uses CoreItem, CoreFolder uses CoreObject and CoreMember and so on.

Update - Things are cooking

I told you I am moving forward... Finished coding CoreID, CoreObject, CoreOwner and CoreMember. Plan to code CoreFile tommorow and then test the components. As you can see, I posting more on the blog... So... the multi-tasking... whats the fuss about? As you know, PalmOS is a single tasking OS for developers (I know that PalmOS itself is multi-tasking but developers cannot spawn new threads - Cobalt solves this but then we know Cobalts problems). I wanted to have a multi-tasking model in Screens so that developers would not have to worry about either single tasks or multiple tasks. They will not need to deal with semaphores, mutexes and other complex operations to keep sync between threads. But the problem was how to make a multi-tasking model which can work on any PalmOS version in theory including Cobalt. Screens does not allow globals or static variables for modules (you use the object storage instead) so it can deal with the request and block of threads via the objects directly. For example: Any module can read a file but if a module wants to write to a file it needs to wait untill all modules that are reading the file, close the file. New read operations wait untill the write operation closes the file. This makes sure that programmers get the advantages of mutexes without having to actually write them. Multi-tasking is the future so its about time that developers not have to worry about the multi threading problems. I'll post more about this as time goes by...

Update - Better than I thought

I am realy moving along... I have nearly finished testing the CoreList component and remove alot some small and some large bugs. OK... the changes: These are the following components to be coded this week: CoreObject - Deals with generic objects. CoreFile - Deals with file stream based objects (you can read/write/copy/resize the stream) CoreOwner - Deals with the owner relationships of an object (parent objects) CoreMember - Deals with the member relationships of an object (child objects) CoreObject/CoreOwner and CoreMember are about 4 public functions each while CoreFile has about 9 functions. These components are very easy and they are the next step at finding bugs in the CoreList/CoreItem components. I realy feel that the train is moving... I have passed the main hard part (the start) and the next hard part is only at the end of the kernel in the multi-tasking part. I would like to thank all the MSN users who are helping me along the way. I couldn't do it without you. OK... So about the open-source bit: The Screens kernel (Core) will be released open-source with documentation and the API (so you can start writing applications using the object storage) . I think it will be GPL but I dont know yet (I need to examine the licence). The reason for this is so that developers can port the kernel to other platforms and the second reason is to show developers that this project is serious stuff. I know many of you will think that the code is small and not very complicated but thats the point. It took me alot of time to get to the simplicity of the kernel or to even agree on the object storage design. When will it be released? No idea... but I think the multi-tasking part should be in about 2 months to be coded & tested. The multi-tasking part is the final hard part in the kernel but is also one of the great parts about it. It will also be very easy to code for the kernel, you wont need to know how to code shared libraries or anything like that. You code a normal PalmOS application but intercept the Screens launch code. You will able to use your normal PalmOS compiler and the API is C/C++ compatible. You include ScrCore.hpp and ScrCore.cpp in your application and call the Core functions. The kernel is quite limited at the moment but the limitations will be removed before I release the kernel.

Update - CoreList coded (renamed from CoreArray)

I finished coding CoreList (used to be called CoreArray). The coding was simple but I was stuck on the definitions at first but a MSN user helped me solve them quickly and easily. I will be testing the component hopefully next week. So what component is next? CoreObject CoreObject should also be simple like CoreList but I might get stuck on the definitions (I am not very good at defining the function names for components) so if anyone can help me via MSN Messanger (zhamilton1@yahoo.co.uk) that would be nice. As I code, things do change but on a very small scale just like CoreList changed from CoreArray. Nothing to worry about since these are not the changes that required me to recode everything like I had to do over the last three years. This is the final code untill the first release. I expect to have CoreObject, CoreFile and CoreLink coded by the end of next week.

Update - CoreItem works

This is it guys... Its going to be the final code for the Core kernel. CoreItem is coded and tested and works! I am now working on CoreArray and then CoreObjects. I am so thankful that I did it because it was hard for me. Development is going to pick up the pace these weeks because apart from a few functions most of the components are based on its previous components and are just normally specific uses of the previous component like CoreString uses CoreFile which uses CoreItem. CoreObject uses CoreItem and CoreArray.

I have decided that I will release the kernel open-source for a few reasons one of them so that developers can port the Screens kernel to other platforms easily. I will release the kernel once its done with the Core API so developers can start learning the Screens coding syntax and start using the object storage.

Update - Things are moving

I finished coding the PrivateBlock component last night. The PrivateBlock is a small component which deals with the way items can have a fixed handle. It doesn't matter now... I hopefully will make an editorial about CoreItem when its coded and working. I am very happy with the progress... I am coding much more now than I ever did for Screens. OK... I think it might be safe to say it now... I am considering making Screens open-source for a number of reasons. It wont be open-source like source-forge in which developers can contribute but developers will be able to port Screens to any platform without a request. So how will I make money? I am planning on doing some kind of Subscription design where you pay lets say 10$ a year and you will able to download modules in a very easy way. More about this when I get round to it.