3/29/01 3-17-04 Spell Cheek, various cleanups and updates. The Design and Function of the Sphere Operating System Alan Grimes alangrimes@starpower.net Foreword The Sphere Operating System is the result of many years of work in the field of Operating System design. This document intends to be a finalization of my ideas about the subject and the concepts I have discovered in its creation. While I now consider operating system technology to be a crude hack that works in place of a more elegant approach, namely a compiler based approach, this Sphere Operating System strives to be the least crude of these crude hacks. Over the course of the Sphere project I have been frustrated by a number of factors ranging from the fundamental complexity of the task to the deficiencies of the x86 PC architecture and the lack of acceptably precise and reliable information about the internal structure of common implementations of this architecture as would be mandatory for the delivery of an acceptably reliable OS product. Also I continue to be frustrated by the absence of clearly documented compiler tools. For these reasons I have been unable to develop a working version of this software. I hope you will overlook this failure while reading about its design. It is my belief that this Sphere Operating System is the only worthy successor to the venerable Disk Operating System. ;) CONTENTS * Intro * The Sphere System * The Core Sphere * Protection mechanisms and security policies * The User Sphere * The Program Creation Sphere -- Interlude * Networking Sphere * Higher Level Distributed Sphere * Application Spheres * A Typical Persistent Storage System for Sphere * A Multiple User System for Sphere * Automating The Operation of a Sphere System. * The Future INTRO - What is Sphere and What is not Sphere Sphere is a system that allows the user to easily manage the resources of his computer. Sphere's primary focus is on this issue of manageability. Issues such as reliability, security, efficiency, and other capabilities are all secondary to this singular objective. The critical issue here is that it not only be possible for the user do all manner of things with his computer, as it is with Unix, but that it also be easy. This text is not about how the internals of the OS work or about the theoretical problems in its design but rather how it should be integrated in such a way that it becomes easy for the user to reap the benefits of these features and capabilities. Sphere is not a filesystem, or even a way of organizing a filesystem. Instead, Sphere is a system for managing and organizing any filesystems that may be available on a given platform. Sphere is not a system for interacting with the user, rather it is a system for managing and organizing a user interaction system. And, as a natural consequence of this focus, it facilitates fundamental improvements in user interface capabilities that are not feasible on other platforms. The question the user should ask is not how things should be done in Sphere but rather how can Sphere be used to solve user problems. I must confess that the range and variety of user problems exceeds my imagination by many orders of magnitude though I often claim my imagination to by infinite. It in fact is, in scope, but certainly not in detail. People say that the Graphical User Interface is the solution to all usability problems. But what if the user is blind? Or what if the user merely has better or more important things to look at? The ugly fact of the matter is that it really doesn't matter how the user interacts with the system. As long as it is conceptually straight-forward, avoiding or hiding complexity wherever possible, and the requirement that the user need to remember stuff (such as key-combinations and commands in vi) be reduced or eliminated, it WILL be easy to use. Naturally, Common sense still needs to find its home in any production interface design but the UI is a secondary concern at best. Note here that when I discuss hiding complexity I don't mean throwing a door over it and welding it shut. Rather I mean compartmentalizing it in such a way that the user can identify and focus on the things that concern him at present. This is important because in many current systems things that are important or useful are buried under and between hundreds of other files and functions and options that are not even related or, worse still, not even available to the user at all. Why should two fundamentally separate applications store their data in the same damn registry? (Creating a massive choke-point for system stability) Why should tools for maintaining (or destroying) filesystems find their home in the same directory as text editors and compilers? Why do programs of fundamentally different classes such as full applications find themselves in the same "bin" with the stupid little utility that does little else but spew out 'Y' all day long? It doesn't make sense and that is precisely the problem that Sphere strives to solve. For the remainder of this text I will focus on methods for designing the system such that the workload on the user is reduced. As there is so much literature about how to make the machine run faster, as well as different literature that doesn't look past the skin of usability. I will stick to the features that my idea brings to the table. - What Kinds of Machines Can Run Sphere? Sphere is a system for organizing and operating 'universal' computers. The design of Sphere is based around the structure of a classic Von Neuman CPU-MEMORY computer that may, but not necessarily!, have input, output, and storage devices associated with it. While theoretically it is possible to get some form of Sphere running on any imaginable computing device, the utility of doing so is questionable. On this class of computers, which includes the x86 PC, there are two fundamentally different methods for delivering crash-proof stability to the user. Whether crash-proof performance is required for your application is another question altogether. Some manufacturers (Amiga) answer this question with a no, I will look at what it takes to answer it with a yes. The first, and currently the most popular, method for delivering crash-proof operation is to implement somewhere between the 'core' of the chip and the actual hardware memory device a system known as a "Memory Management Unit". This device implements one or (hopefully not) more methods for partitioning the available memory hardware among one or more virtual machines. Such systems allow the programmer to write arbitrarily erroneous code without any risk of doing damage to the rest of the system. There are no restrictions on what the programmer can write or in what languages he can write it in. The cost of this system is the overhead and cost of the MM hardware. Even with modern paging techniques, memory still tends to become "fragmented" by successive allocations and releases of memory. Due to the nature of these languages it is not feasible to "defragment" a randomly populated address space to allow the insertion of a large data structure for which there is enough total space but not enough contiguous space. Another flaw in these systems is that it often requires great amounts of time to switch from one protected area of memory to another, a process known as a "Context Switch". The second method is to require all compilers on the system conform to a strict set of standards. The key feature of such compilers is that they don't support unrestricted pointers and are otherwise "safe" if not Type-Safe. That is all object references must be to an existing object which has already been assigned, by the compiler infrastructure (an OS by any other name), a position in memory. On MM based systems such an erroneous reference can only cause erroneous operation of that one program or trigger a forced V.-shutdown. This alternate system works by virtue of the fact that all symbol allocations are made through the OS itself. When a symbol is allocated the OS will add that allocation to the list of memory owned by that process. Should it be necessary to move an allocation, the memory management software simply updates the symbol tables. On such a machine a special illegal address could be set aside so that the memory management software can trap the exception and then use it to implement a virtual memory system. Such a system could avoid all the overhead involved with operating a MM and multiple virtual machines. The best example of a system that works along these lines is the superlatively brilliant Squeak, a Smalltalk 80 system. It is available for many hardware, operating system combinations and can be found at www.squeak.org . You are probably wondering how two vastly different schemes such as this could be implemented by the same OS. The key here is that what is properly called the "kernel" in other operating systems is a minor component of Sphere and completely optional and replaceable. These memory management functions are invariably handled by a device called "memory" in the sphere system. Each is simply an alternative approach towards implementing memory management. While it's doubtful a system developed based on one would be completely compatible with the other (at the level one normally thinks of compatibility), they both implement the same *system*. This means that unlike other OSs, Sphere is not a realm of compatibility, rather it is, in the purest sense, a System for operating computers that may be seen in many (or no) implementations. - Design Terminology and Methodology One thing that is common to every OS text I have seen is a chapter devoted to the history of operating system technology. Its as if the author is trying to make an excuse for how much today's systems continue to resemble or are even directly derived from those that were developed and deployed thirty years ago. In developing Sphere I did not look to history or endorse the continuation of a single course of evolution from the time before I was born but rather I looked closely at how computers are built and used today. You have seen that I have chosen the word 'Sphere" to describe my design. Let me assure you it took me a lot of crazy and stupid attempts before I settled on something this elegant. My choice was made as I realized how powerful it is as a metaphor for what happens within the OS. This isn't to say that this is the One True Metaphor or that it should be promoted where others are more applicable to the specific case, Just that it works very well. The dictionary definition of a sphere is the collection of points in thee dimensional space that is equidistant from a point called the center. Sporting equipment such as basketballs closely match this definition. While it could be argued that the Sphere Operating System deals mostly with the outer shells of software and their interactions, software itself is far from hollow. To accommodate this I will adjust this definition slightly to mean the volume that is less than some distance from its center. It is interesting how this maps to today's computers. While imagining what a sphere would look like in more than three dimensions is difficult, having several potential representations, It is quite easy to cut sphere down to smaller dimensions. Cutting a sphere in half (or along any other section) leaves you with a circle. Cutting again gives you a closed interval along a line. It is this most primitive level that our computers currently operate. The memory within all conventional computers is organized as a collection of discrete cells that are accessible by numerical values from some low value, to whatever the upper bound of available memory is. Even though it is discrete rather than continuous, it is still referred to as "linear memory". Another famous theoretical device, the Turing machine, also uses a linear memory which is weaker than the conventional version as it cannot be accessed randomly. Still, we can discuss a computation on a universal turing machine which takes finite space on the tape, a version of a sphere. Attempting to implement a turing machine program for my word processor would result in a tape of fantastic length and complexity. But, if we were to split the functions into a collection of turing machines along their natural borders such as networking, disk, user interface, etc we not only make it much easier to understand and program but also reduce the total number of cells of tape used for the entire computation. Seeing how each of these focused turing machines also operates on a sphere of sorts reveals nearly the entirety of my design. While I can only guess the shape of computers to come I can flex this metaphor some more and show how it can be used, to great effect, as a weapon against complexity. I will leave the details of the system to the next section. Here I will round out my look at how systems works and complete my metaphorical toolbox. One of the easiest ways to view a system is as a discrete entity is through a modern network such as the internet. A system on the internet is a computer that, As I've shown, can be represented as a sphere that interacts with the world through its network connection. That is it provides or receives services over the network. It is the provision of services that is of greater interest to the Operating System designer. A modern internet server may consist of several or even dozens of machines working as a team in any of a number of arrangements. The factors that go into the design of such services are outside of what I am ready to discuss here. The feature that is of interest to Sphere is the concept of the network connection. The most popular connection protocol is Transmission Control Protocol (TCP). The point is that clients can *connect* to services and then, with this connection, carry out *transactions*. I am using the term transaction in its broadest sense that is to mean the unit of service that is provided by the server. A transaction can be the delivery of a webpage, the setting of an airline reservation or simply getting a file via any type of file transfer protocol. In Sphere, the system handles connections while the actual spheres handle transactions. The concept of connections aren't strictly necessary but they make the system operate much smoother by providing a mechanism for notifying parties in a connection of failures/'hangups' on either end, they also provide a means for servers and users to monitor the system and understand its shape. The shape or topology of a Sphere system is a function of its sub-systems and the connections between them. These connections can form a graph with the spheres being the verteci and the connections the edges. A discussion of what a graph is can be found in a typical discrete math book. It has come to my attention that Unix too has a version of connections called pipes or sockets. Unix sockets are weaker than Sphere sockets in many respects. First and foremost, is the plain fact that there is no way for the *user* to make simple queries about the structure of his own system. There is no obvious way to determine what services a given daemon program provides nor is there any direct way to inspect the graph of connections between clients and servers. These deficiencies, apparently, seem unimportant to the developers of Unix but they do have very important consequences in terms of what the user is able to do with his system. Because the user cannot detect the topology of his system he is greatly impaired in his ability to determine what might be causing and correct behaviors that he may not understand. An obvious practical instance of this is the almost universal experience of starting an X-Windows application with an X-term and then watching in frustration as it's separate window disappear as you close the said X-term. To the novice user, his only recourse in this situation is to start a new X-term for each application and then minimize its window after the desired application is started. Equally important is the user's inability to discover and use advanced features of his system. Most notably, he is unable to determine which named pipes (or even what files are, in fact, named pipes) are associated with which running process, whether the process associated with a given pipe is in fact active or just about anything else without the use of tricks that I have not yet learned after three years of continuous Unix usage. In fact, Unix makes a policy of dispersing related information as widely as possible with the effect, if not the intent, of making it virtually impossible for the user to discover the relationships that exist between the various pieces of software on his system without great hardship. Experts will quickly point to more efficient uses of X-terms but the issue is that expert advice should _never_ be required to use one's computer. Furthermore, experts more knowledgeable than myself will be able to point to where one should look for information on how to use the named pipes that may be available on one's system. Again, the fact that the system itself does not provide the answers is a major but overlooked problem. The rest of this text will describe how this system is applied to the common problems operating systems are usually tasked with. THE SPHERE SYSTEM In the last section I discussed the Official Version of the genesis of Sphere. Here I will look at how this metaphor is applied to the design of software. In designing a very complex system it is often more than useful to break it down into easily designed sub-systems. In such cases where the subcomponents themselves are extremely complex it may be desirable to apply the process of subdivision again. When this is properly implemented, the Sphere Operating System is, indeed *recursive*, The concept of recursion is a common theme in computer science that has not yet been exploited to its fullest in the design of systems. A sphere in the Sphere Operating System shares almost all of the properties of the computer it operates on. This may lead some to call it a virtual machine operating system. It is a V., of sorts. Whether hardware support of virtual machines is utilized is an implementation issue. However, in practice, an implementation of a distinct and largely independent environment is rarely desired. The only real reason for such an environment is to support legacy or experimental systems, or perhaps even for migrating from older versions of Sphere. Sphere is not, therefore, a V.-OS, rather it is a System Oriented Operating System. A SOOS is more powerful than the traditional Unix process model but at the same time not as strict as a V.-OS. Just as the machine as a whole must be prepared to accept input at any time the user wants to type or some other event happens, each Sphere within the Sphere Operating System must be able to accept asynchronous input through what is, in effect, a type of Remote Procedure Call. This is entirely at odds with other operating systems in which a process must use sophisticated methods to talk with each other, or rely on methods such as synchronous pipes that are provided by the OS itself. In Sphere all the mechanisms necessary for synchronization are available internally to each module such that the user of a given sphere's services doesn't need to know about what synchronization methods it employs. The actual nature of the synchronization methods available depend on the execution model used by the system. In some interpreted systems, the system is run statement by statement or function by function and the CPU device will invisibly enter each sphere and retire any and all available statements for as long as the current sphere has execution resources remaining. In explicitly threaded systems things are a bit more complex. In such systems, commands for manipulating the scheduler are provided and even commands for implementing a custom scheduler for each Sphere! Another interesting property of Sphere is that just as easily as you can say all of main memory is the outermost sphere, you can just as easily allocate as many Spheres as memory permits. This is a natural property of the system. It may not seem important but in some hardware configurations where memory is distributed among multiple semi-independent computers, also known as a "multicomputer" it becomes quite useful. In such a situation, a global sphere can be created and a second layer of spheres can be used to wall off address spaces. By altering the driver set available to each sphere, devices can be associated with processors and a skilled user can arrange systems within spheres to meet his performance goals. Partitioning the system can be accomplished by disconnecting spheres or groups of spheres from their peers. Management of distributed services across networked systems will be discussed later. A silly thing to do with this capability is rig a normal PC with two Spheres and do split-brain surgery on it, having half a load of drivers in each. This ability is not really important except on high-power implementations. Partitioning systems for users and other entities will be discussed later. One of the drawbacks to my design, however, is that it is highly demanding good coding practices. Where a minor flaw in a Unix application may never cause any problems because processes are highly transient objects. Spheres, however, are intended to remain in memory and function correctly until they are no longer needed. Problems such as memory leaks and irreversible behavior (such that the sphere cannot be returned to it's ready state) would be far more inconvenient on a Sphere system than on Unix. For example, if we had a sphere which provided file archiving services using various compression algorithms. It is desirable that these facilities remain operable indefinitely and never have to be reloaded regardless of how many transactions they serve over their lifetime. THE CORE SPHERE A system such as Sphere requires a minimum set of 'core' functions to provide basic services. It is desirable that these core functions be provided by a Sphere to the rest of the system so that the system itself is reflexive. Naturally the name CORE is used to label this provider of Core services. The first step in discussing the requirements for this Core Sphere is to demonstrate the necessity for its existence. A good starting place for this endeavor is in these funky post-modern TV sets they are producing these days which sometimes have configuration menu-trees so large as to be mistakable for the directory tree of a large, poorly organized, mainframe. This menu system as well as many of the picture manipulation functions is doubtlessly implemented by one (or several) embedded computers. The relative power of these embedded machines is not important to this discussion of the software that runs on them. Nor is it my intention to present Sphere solely as an embedded OS. The requirements of a general purpose computer are that it be able to take input, run almost arbitrary programs, and provide output. That is it must be possible for the user to sit at a console of some sort and input arbitrary programs that the computer is to run. It is also desirable for these programs to have as little information about the actual machine that runs them as possible so that they can be moved ("ported") from machine to machine. It is also desirable to be able to load and run many such programs concurrently and in an organized fashion. My argument is that to meet these requirements with conventional technology using the Sphere system a Core module must be implemented. The programming on the embedded system within the TV may consist of several sphere-systems, some for the code behind the menus other completely separate ones for processing the video stream, and finally, one to retain the user's preferences. This is using the broad definition of spheres rather than the proper one defined by this text. Our interest is in the user interface and persistent setting-storage spheres. The first undesirable feature these software systems have is that once integrated they cannot run on any other TV-machine, even on TV-machines that have equivalent functionality. It is quite possible that a TVmaker will use the same code to operate a whole line of products, only changing the set of drivers included in the binary. Even at the level of linkable object code, there are problems with namespaces. Say a change were made to the system such that the user was given a choice between the menu of 1001 options and "Just give me TV, Damnit!" The code to do this would be something like: ------- READ "option" Switch to "option" 1 : MENU-INSANITY 2 : JustTV ------- The normal operation of either system requires that they be implemented with a collection of symbols and that they have access to symbols provided by the system such as computational op-codes and environments such as the drivers and library functions. A major concern is where the set of symbols implemented in Just TV 'intersect' and Menu Hell. In this case the more apt term is collide or crash. These "multiply defined" symbols will cause the linking process to fail. There is no reason for the wise and noble implementors of Just TV to be limited by the naming choices in MENU-INSANITY. These are merely alternate versions of the same system, in Sphere they would be interchangeable parts for the same metaphorical socket. No such sockets are implicit to the Sphere design except the core sphere itself. In addition to just switching between menu functions, our meta-menu must now also do something similar to activating the symbol-set used by each program. This requires that the environment that runs the system allow programs, such as the meta-menu, written for that environment to request alterations to that environment, that it's request changes in the interpreter's dictionary or symbol table. Now lets suppose that the machine driving this TV device is a single processor that must both decode the video in real-time and, when called upon, perform menu functions as well. A design in which the menu can, potentially, take any amount of time before explicitly returning the processor to the video-decode function would not be able to guarantee smooth operation of the video decoding. his requires that after the meta-menu has chosen a menu system it must then ensure its operation does not interfere with other functions of the system. While the time requirements of this video decoder are highly predictable, it is still not desirable for the menu-author to have to consider this in designing his menu because he wants his menu to operate on other setups which may have different limitations. For the video system to work it must be able to insist uppon a certain ammount of processor resources at certain times. It is also desirable that this system be able to split the remaining processor time fairly among all the remaining tasks that don't require real-time response. What we require is the ability to create multiple 'execution domains', a subset of which is the popular technique known as 'threading'. Whatever the execution model, the idea of sub-dividing computations and then assigning them different processing resources remains the same. That is the environment needs a way for Spheres (or components thereof) to request alterations to how the environment processes them. Finally the most fundamental resource of all, memory, needs to be controllable from within the system. Environments such as Java will let you allocate memory and, like a good butler, clean up after you are finished with it. For the purposes of our example the user needs to be able to create objects, such as new spheres or new data allocations as well as cleanly destroy ones that really irk him such as the reviled Menu Insanity! Because all of these services need to be provided to the system and, by definition, all accessible memory (including firmware) is a part of the global sphere these core functions need to exist within it. Other systems make this global sphere very complex, incorporating these functions and many more. This is a suboptimal approach because it is extremely complex and fails to provide any way for these functions to be applied to the system itself. Sphere's approach is to be reflexive. The Sphere Operating System is itself contained within a sphere called "CORE", a peer among all other spheres running directly within the "GLOBE" sphere. Now we have a rationale for putting the system within itself. The reader has a fairly good understanding of what this core sphere does. The remainder of this chapter will explore the concepts and techniques used to design an implementation of the Core Sphere. Okay we have a spherical system and within it a sphere designated CORE. On a well built system the user will never have any compelling reason to study the inner workings of the CORE sphere. At most he will notice it in a listing of Spheres on his system while he is engaged in some administrative task or another. To him this sphere will be known only by the services it *provides* to the global sphere. Any sphere can provide services in this manner. Because of this the first set of services provided by CORE to the GLOBE involve the publication of services. CORE therefore has no truly unique role in the provision of services. This functionality relies uppon the existance of a data structure associated with each sphere, including the GLOBE. I will term this the Sphere Data Structure. It can be implemented as a class in some object oriented language. The exact format of this data structure is implementation specific and under no circumstances should be revealed to the user. Its mere existance should only be a point of conjecture to anyone who doesn't have access to a manual. Allowing it to become visible, like 'I-nodes' sometimes are under Unix, would cripple the CORE module's ability to abstract the system, CORE's primary function. CORE is therefore the set of code that turns the SDS into Sphere- objects. In addition to the provision of the Sphere abstraction, which I will discuss shortly, CORE also provides *the facility for providing* suitable standardizable abstractions for the varrious pieces of hardware that may be peripheral to the CPU. These may include Auxiliary Processing Units, as well as all the types of hardware one can find on contemporary PCs (inclusively). This is implemented within the CORE Sphere as sockets for both *devices* and *interfaces*. A good implementation of CORE will not accept the addition of any code that is not a device or interface. The only two devices that are absolutely mandatory to the operation of CORE, and must be loaded into CORE before execution begins, are CPU and MEMORY. These two devices must export classes for manipulating program execution and memory protection respectively to be inherited by the SDS. Whether the system supports hot-swapping of these objects is an implementation issue. CPU and MEMORY may be implemented by a conventional kernel or a compiler environment. Wheather the driver architecture is implemented such that CPU and MEMORY are a single device-driver with two "functions" or as two discrete drivers is unimportant. Using these two inherited functions CORE can provide additional services to GLOBE involving the creation, manipulation, and destruction of spheres. Once a sphere is started and a thread is started within it, that thread can only manipulate sub-spheres. This is for security reasons. To gain additional privileges, the user interface sphere must gain access to a sphere higher up in the hierarchy. This is discussed further below. Inside CORE, an additional set of functions are provided for manipulating drivers and interfaces while other global functions are blocked or over-loaded to restrict what can be done within CORE to what should be there. The device driver architecture that I prefer is to have the drivers exactly reflect the chipset as implemented in the machine. Each functionally significant chip in the chipset, or ensemble thereof, should be represented in the drivers. They will then export whatever interfaces are available on the current implementation to a list, maintained by CORE, of such interfaces. Because it is uncommon to find hardware by different vendors that support exactly the same interface, a second class of interfaces will be provided by another set of objects. Lets say a certain chip implements a clock of some sort that provides a non-standard system for measuring time. An interface called TIME may then be implemented that acquires and locks CLOCK and then provides the standardized TIME service to the system. These services are provided to the GLOBE by a small set of interfaces that lets other spheres in the system connect to CORE, query the available devices, and then acquire and lock direct interfaces to them. A filesystem sphere would work by acquiring and locking the block devices it is to manage. In special cases where systems such as the video decoder of the TV mentioned above need special access to the CPU they will be implemented as uninteruptable devices in such a way that they can lock portions of time from the CPU device. Other pieces of software which have special relationships to the CPU such as guest runtime environments such as Java may be implemented as an interface to the CPU. A Java executable, for example, would be loaded and run as any other sphere. It would be interpreted by an extension to CPU within CORE. Each executable format will have an interpreter of this nature that is responsible for any bindings necessary for it to communicate with the other spheres in the system. The fact that the CORE sphere must be implemented without requiring any services not provided by CPU and MEMORY does not cripple the system. Devices such as "ERROR" or "CONFIG" can still be implemented in general and versatile ways. Under a default configuration ERROR would operate on a hardware debug console, and CONFIG may point to a CMOS memory. In other designs the user would rather have errors directed towards a virtual console and the configuration stored on disk. Even with the limited information allowed to CORE this can be done quite elegantly. When the user system is available the ERROR device can be replaced by a mischievous piece of software which calls services provided by the USER sphere to annoy the user with useless pop-up messages. Strangely, such implementations seem quite popular. The CMOS V.s. disk switch can be accomplished similarly. PROTECTION MECHANISMS AND SECURITY POLICIES It is important for any modern computer system to be able to maintain its own integrity against deliberate attacks as well as allow the user to identify and eliminate malicious or unauthorized software that may be running on his machine. Other systems use highly contrived and arbitrary methods usually introducing many unnecessary and ill-fitting concepts into the system. Unix relies on the honoring of certain "protection bits" and user ID's. Unfortunately this requires an extensive collection of utility programs as well as a large amount of user effort and expertise to maintain. Other systems designed without security in mind now have to retrofitted with solutions focusing on points of entry and third party utilities to reverse the effects of intrusions as the only means of protection. In Sphere, the very architecture of the system is exploited to deliver a secure computing resource. This is guaranteed by the rule that no system command is *ever* interpreted outside of the sphere which issued it. Someone might think that it would be trivial to write a sphere which contains a command to delete the CORE sphere. If he attempts to implement as such, it will indeed be interpreted by code in the CORE sphere but it will not actually affect that sphere in any way, instead it will be interpreted *within* the calling sphere. If the calling sphere contains a sphere called CORE, even an instance of the actual CORE itself, it will be deleted with no effect to the rest of the system. Otherwise it will simply fail because there is no sphere named CORE *within* the current context. However, it is possible to gain a higher level of access by issuing a request for such access with the necessary system key. Presumably this request would be digitally signed with said key providing the necessary safety from forgery. The request would have the effect of creating a virtual sub-sphere within the current context that points to the external sphere. The current context would be able to then execute what would otherwise be privileged operations on it. By this mechanism the user only needs to make sure that no privileged context is available within the context he is working in to guarantee that his system is not at risk of damage by his programs. Other systems would require a review of the permission of each of the, on average, 30,000 files that are usually present. THE USER SPHERE Any Sphere system that is to support user interaction beyond a possible system maintenance console should deliver a user interface sphere. This sphere should acquire any and all user interface devices associated with a physical workstation from CORE. In this usage scenario, the system is strictly single-user. It may then use a key to gain privileged access to the system if system maintenance is required. The User interface sphere should only be standardized in how it interacts with the rest of the system. Nothing should be assumed about how information is presented to the user after it has been transmitted to the user interface module. This module is here to give the user ultimate control over how information is presented to him. Versions of this sphere could be the engine behind everything from a neural interface to a simple teletype terminal. No implementation of this sphere should support any method for querying its capabilities or settings because it is merely a conduit for communicating with the user, a translator if you will. If its workings were open to the dark side of the source all kinds of stupid evil limitations could be imbedded into other spheres that have no business knowing that much about how the user likes to operate. It would limit their usefulness. The User interface sphere may also provide facilities to run commands like any other shell, though any number of techniques can be employed by this sphere on any number of devices at any time, which may or may not include classic utilities, or may accomplish the same ends through data "workbenches". Various User Interface Spheres may implement any number of types of sockets for such plug-ins, anything to make the user's life easier! Most of the unixy type data manipulations would be accomplished through that type of plugin. Advanced application-system spheres such as Corel WordPerfect Suite, StarOffice, Netscape and other products. should be implemented as full spheres within the User Interface Sphere, as such they can have more knowledge about the workings of the UI than other spheres in the system and can therefore have much greater interaction with the same. These are considered "User Applications" I will discuss "general" or "traditional" applications later. I have done very little work with application development due to the extreme and pressing need for a better OS. Since this new OS would make studying existing application programming interfaces pointless. I hope to avoid having to put up with the mindlessness one sees in all contemporary operating systems and focus instead on developing and promoting this project. THE PROGRAM CREATION SPHERE To write and test programs a safe, effect-free environment is needed. A sphere solely for program creation and debugging is an appropriate solution. By providing dummy drivers any sphere in the system, including CORE, may be inspected and debugged in a safe fashion. In addition to providing a test environment debugging spheres also provide a workbench of integrated and interactive development tools. To support this the operating system only needs a facility by which it can block any system calls from recursing down to the underlying system and then redirecting these calls to dummy handlers. This can be used to test device drivers by replacing the IO primitives with a suitable device emulator. How the debugging suite is implemented and what capabilities are available depends on what kind of system it is running on. In networked systems, any code downloaded from a website or in an e-mail should be run in a similar way to protect the rest of the system and to test for any possible 'trojan' behavior. Historically a number of operating systems have been shipped without compiler tools or shipped with ones so weak that they are incapable of running anything more than toy programs. Any complete distribution of the Sphere Operating System will provide the user with the ability to rewrite any sphere or loadable extension in the system. -- Interlude -- This text could end here. I could implement a marginally better than degenerate version of the software described above on a 32k cartridge for an Atari 800 and call it a workable system. You will, no doubt, note that no such system was implemented for that machine. That machine was so simple that it didn't NEED that system. It didn't need an OS at all. Since the parameters of its operation were so limited, it worked like the TV above except it supported program switching by physically swapping cartridges. I was in a library this very day wondering how the notion of operating systems have changed over the years. I came across a text printed in 1981 that described about 75% of what I am proposing here. (I have since lost the reference, it was only a few pages). The concept is still good. The problem is that people tend to stick with what works, what gets the job done. Existing systems, particularly Unix, continue to work for most things and so people continue to use it. I have met numerous fanatics who say things like "E Pluribus Unix", as a rallying cry. Which means, roughly, that all operating systems not only will but should converge to one single standard that will bear great resemblance to one of the current versions of Unix. If that were to come to pass I can say I will not be a computer user, rather some poor schmuck who depends on someone else to admin a shell account for him. If I'm lucky I might manage to maintain my own system only to the point of being able to surf the web but not be able to reap any of the potential rewards computing has to offer. There is a real problem here. While dozens of efforts have been made to solve the problems of making Unix work acceptably well with the latest hardware absolutely no work has been done on the human brain to make it work better with Unix. That's a little joke, laugh -- at Unix because the joke will not be on me! The truth here is that today's machines are millions of times more complex than that Atari and they're getting more complex at an exponential rate. The systems we humans will be confronting ourselves with in a few years will have many gigabytes of memory and terrabytes of storage. They will run thousands of programs at once and humans will have final responsibility for keeping them running. While the very best of our minds can still operate the current generation of computers, what of the next? Will humans be reduced to a "device which types when issued a read request"? Or will they be masters of a whole new universe of possibilities? If I have any say in the matter, it will be the latter! Sphere will make it possible. I don't claim that Sphere is unique in this regards, only that it is my personal attempt at a solution to the problem. NETWORKING SPHERE A major requirement of any contemporary system is that it be able to communicate over a network and support the popular network applications and functionality. As almost everything in sphere, the network will have its own system sphere. It will be able to connect to low-level network devices in the CORE sphere. It will also provide sockets for drivers for all of the layers of the appropriate network protocol stack. It is unlikely that these layers will be active, rather implemented as passive bits of code, encapsulated by spheres, which threads pass through in the course of their operation. Only the most trivial of applications, such as PONG, will actually run within the NETWORK sphere. Other applications will connect to the network sphere and acquire the resources they need. Smaller user utilities will operate under the USER sphere. Network configurations may be saved through the configuration storage driver in CORE or by use of a filesystem Sphere. As I have shown in the section on user interaction, the user can "enter" the network sphere, inspect, and modify its structure. It is this accessability of the inner workings to easy inspection and modification that will allow Sphere to surpass its predecessors in terms of usability. While it is theoretically possible to inspect and modify the code of a system such as Linux, it has already grown far beyond what is feasible to the casual user mostly because the system makes no attempt at all to reveal, let alone present, its own structure. Attempts such as the Linux Graphing Project have yielded some interesting results but remain unusable for anything beyond the kernel. In the grand scheme of things, the Network sphere is primarily a service provider. It provides services to other spheres which do the interesting work. It is what gets built on the network sphere that really puts Sphere in a class by itself. HIGHER LEVEL DISTRIBUTED SPHERE With networking subsystems available it becomes possible to contemplate a distributed platform environment. This environment would be implemented as a sphere that, through the network, connects to its counterparts running on other machines across the net. By running a standardized internal environment, across all implementations it can create a purely peer-to-peer distributed platform. The distributed sphere is only a vessel for running higher level applications. As such it only concerns itself with maintaining the network and providing communications channels to applications. This arrangement of having a distributed OS running on-top-of (or more precisely, next to) a totally network-ignorant host OS is the ideal for such systems. Its great triumph is that while other operating systems such as Amoeba, Plan 9, QNX, and others incorporate distributed computing at a lower level and therefore aren't well suited to working with systems outside their own little domain, Sphere on the other hand can run spheres for every one of them simultaneously with no problems. I must give Chorus some credit here, it would seem that COOL-2 is very close to what Sphere is from the ground up. Not worrying about distributed computing outside this sphere allows the non-distributed spheres to avoid any performance hits that a distributed object system incurs. Other more famous systems such as Gnutella and Freenet are weak in comparison. They seek to be merely filesharing or communications networks. Sphere's main distributed computing system is a general purpose server that can be tasked with any purpose, including sharing files. This generality is merely another manifestation of the underlying philosophy of the Sphere System. Additional comments about this sphere can be found in an article I hacked together back in '98, available on request. The Spheres described in the following sections may benefit from running within this environment to varying degrees. APPLICATION SPHERES Major applications such as databases or virtual world systems (similar to MUDs) and other service providers that have a purpose that doesn't directly involve user interaction belong in their own sphere. The potential scope of such applications is so broad that discussing them in detail is not practical. The sphere system also, being well described, doesn't warrant additional description. What remains is how such application spheres should interact with the system. A first thing all such applications should do when initialized is to check the system to see if it is already running, perhaps on another processor cluster? If that is the case it should synchronize with itself and set up a proper division of labor. Applications such as databases may, for performance reasons, prefer exclusive access to a mass storage device. Other special applications such as a factory robot controller may, at this time, gain control of the actual hardware and the worker's console. On such a platform, it may indeed be the only module in the system outside of CORE. When initialization is finished and all external resources are acquired, the next step is to publish an interface and go into business. There are two fundamental ways of doing this. The first is to publish your list of calls directly. Since this may make the list of functions at the Global level overly large, a better way to do this may be to publish just a function by which clients may request from you a specific interface. In a situation where technology has moved on to FOO-2 but yet a large number of FOO-1 clients must be supported, your software should publish the new interface by default and then give the user the option of activating the old one if the need arises. The longevity of an installation should be taken into account, as a single piece of software may exist unaltered on a host for several decades, it would be most silly for your module to continue publishing FOO-1 when its only being kept around to support FOO-2 because everyone else has moved on to BAR. A TYPICAL PERSISTENT STORAGE SYSTEM FOR SPHERE So here we come to the persistent storage sphere. Some people say that the efficiency of the persistent storage subsystem is the most important part of the OS. In Unix, the filesystem is the centerpiece and all other software is twisted and perverted to revolve around it. Unix forces the filesystem to do everything, therefore it does everything poorly. In Unix the filesystem is security, user separation, interprocess communication user grouping, and device interface. If these functions were removed from the filesystem, or rather the filesystem removed from the above design goals, they could be pursued in a much more rational way. Another group of existing systems go to the other extreme. They organize the system around the software, which is usually the best decision, and then make the mistake of linking it strongly to persistent storage; they make everything persistent! This is the other extreme. It leads to counterproductive disk writes and disallows the user from dealing with the filesystem as a distinct entity. This is bad because it masks a fundamental distinction between different types of memory. That is, the primary memory can be directly accessed by the processor and the second can't. Certain implementations of "memory" may implement virtual memory but the machine starts from a clean slate every time the power is cycled. In Sphere the filesystem is treated as a software system built upon various devices and then provided, as a service, to the system. It has no special rank or position in the Sphere system, In fact, for all implementations of Sphere that I would consider correct, its existence is completely optional. It is just as possible to load programs from over the network as it is through a local filesystem, or even export them directly from the development environment. Having a filesystem is merely a *convenience*. In theory a system could function for an arbitrarily long time and never bother with saving stuff. It is only flaws in today's software and dependance on external power supplies that forces a system failure and subsequent reboot. The structure of the filesystem sphere is very much like the network sphere. That is it supports subsystems for each of the functional layers. This is a departure from existing systems which treat everything between the block device and the user as one single system. In Sphere the Persistent Storage System sphere recognizes that there are three things going on. Once a block device is acquired from the CORE sphere, an allocation policy must be enacted. Then these allocation units must be named by an Object management system (possibly several). In addition to the basic operation the filesystem sphere must also provide for high-level maintenance and repair of the filesystem. It is important to keep these layers separate as user choice is the overriding concern to sphere. To keep things simple I will look at how DOS FAT 16 would be supported under Sphere in a rather simple and generic Object System sphere. More advanced systems may extend or replace this design. A disk allocation policy is a means for dividing blocks of data among users. Unix implements a fairly elegant system. DOS's main aim was to compensate for the unreliability of the drives available at the time of its creation. It therefore maintains two copies of the File allocation Table. Its design is irrelevant except that it accomplishes only one function, to divide the available blocks into clusters which can be allocated in the form of chains. Regardless of how sophisticated your filesystem is, this is its most basic function, allocating blocks to clients (programs). DOS's naming system (ideally) is implemented as special files registered in the above allocation system. These files contain lists of "Directory Entries". These directory entries contain information about other objects on the volume including files and directories. Using this system it becomes possible to uniquely identify files on the volume, volumes are in turn named for easy identification. If this volume were accessed through Sphere, this document would be tagged: zoom:\prog\os\doc\sphere.txt Another example would be (this time a game demo CD from 6 years ago): buyer_s_gui:\bioforge\install.exe The creator of this CD wasn't very careful in labeling it but the system is not fatally flawed. When spheres are loaded into the system, they may be issued access to only a certain path (including paths linked to that path). This is done for security reasons. It allows for quick un-installation and protects the integrity of the permanent system image from damage by bad software. This method of protection is typical of what you will find in Sphere. Applying this system to object management systems that are *gasp* innovative such as one proposed for ReiserFS will be a very interesting project. But first I need to put Windows 3.11 in my past before I will be able to work on that problem. A MULTIPLE USER SYSTEM FOR SPHERE Nothing prevents the implementation of a sphere for managing users, or as they are called in Sphere "entities". Entities are either users or organizations which work by offering links to its members. Unlike numerous UNIX implementations which can support only one group per user, there is no limit to how many organizations can be linked to from a single user sphere. The Multi-user sphere handles logins and maintains "console sets", sets of hardware that become accessible when the user logs in at a certain terminal or network connection. This module maintains user generated objects with the persistent storage system in strongly encrypted archive files. After login a user system' sphere is started and given access only to what the user is permitted to access. All modifications a user attempts to make to the system or system files are re-directed and restricted to his own sphere. If he tries to modify the keyboard setting to DVORAK the change will be saved to his own configuration files and affect only the operation of the keyboard while he is logged in. In this way the Multi-user sphere works much like the program creation sphere. Collaborative work is possible through organization entities. The user, from his sphere can enter the organization sphere and then start or access any spheres that can be loaded into that organization sphere. Should two users be accessing that sphere at once, it will allow instantaneous synchronized communication between them for any application, depending on how well that application sphere was implemented. In my opinion this is the one true way to go about partitioning a system for users. It's self contained design allows system designers to include or exclude multiuser functionality as circumstances dictate. Furthermore, it allows programs that are not directly a part of or related to the multi-user functionality of the machine to ignore its existence. AUTOMATING THE OPERATION OF A SPHERE SYSTEM Throughout this text I have described a system in which complexity is hashed into a handful of Spheres which may be subdivided again as desired and appropriate. I have discussed how these spheres may export an interface through which they may be accessed. This combination of "divide-and-Conquer" and domain-specific interfaces yields a system of unprecedented manageability and high potential for usability. Presently, in this last descriptive chapter, I will deliver the Coup DeGrace and propose a method for delivering a level of user interaction that hasn't been dreamed of for nearly twenty years now. What is interesting about a system design that follows such a well ordered division of labor with a set of standardized commands throughout, Or at least a "Machine-Help" call where customized abstractions are used, is that it becomes such that it can be modeled quite easily by an automated agent. In such a regular well-ordered structure, a very straight-forward operator program can be written. In fact a sufficiently powerful program was developed back in the late '70s. Its name is SHRDLU. SHRDLU is a simple AI program that manipulates a world consisting of blocks and boxes. It can do this because everything in that world is represented by symbols. Since Sphere also meets this requirement SHRDLU can, in theory, be sufficiently intelligent to operate and manage a Sphere system. With such an agent at the user's disposal, all but the most basic training requirements can be avoided, even for users of very complex systems. To such a user a suitable AI can act as guide and servant. If the user desires a monthly report on the operation of the system, the AI can tour the system and query each component for the desired information without the user having to do anything more than ask. By knowing all the rules that apply to the operation and modification of the system the AI can act as an expert advisor to the person tasked with running the system. Having an AI at hand allows the user to simply explain to the AI what system policies he desires and expect the system to arrange itself to those requirements. It doesn't matter how large the managed system becomes because the basic rules remain the same. In discussing AIs I must also remind the reader of the limitations of the conventional AI systems. AI systems are still very primitive. While they can be of some service, their capabilities are still extremely limited. Perhaps we should be thankful for these limitations but I would still like to see more. THE FUTURE So there you have it, the bulk of my vision for the Sphere Operating system. There is one question that remains to be addressed. Where does Sphere fit in the grand scheme of things? How will it evolve to meet changing demands? What is Sphere's role in shaping the future? Sphere is merely a next step. A rehash of old ideas into a new synthesized whole. It is not perfect nor everlasting. It is better than the system from yesterday yet not quite as good as the system of tomorrow. It excels at solving some problems yet doesn't solve others. Beyond Sphere lie High level language based systems such as LISP/Scheme and Smalltalk/squeak. Such systems, with a designer more clever than I, can be made to solve many of the problems Sphere does, better even! As such a system would be an even more radical departure from what is the widely accepted standard I see Sphere as a logical next step, a bridge to such a future. My experiences with squeak show that it, itself could benefit from some of my sphere concepts, especially in the areas of system subdivision and security. In fact, a group called "Squeak-e" has created a class called "Environment" which could be considered as a separate attempt at implementing Sphere. Beyond these high level language systems is the ultimate end of artificial intelligence, or what I prefer to call 'cybernetic intelligence.' Once that is achieved the world will be forever changed. Neural interfaces become possible. Even in such a radically changed world the humble old Sphere Operating System may find a new role as a bridge between the chaotic and colorful world of light and sound and the rigorously ordered system of databases and text messages. In such a use Sphere must be as responsive to thought as my hands are while writing this. It is my belief that it, at least, has this potential. It strives to be powerful and interactive where other systems are mute and confusing. Because the system does not revolve around a single immutable kernel, it will always be possible to add the overlooked subsystem at any time, without even rebooting the system. QNX has "it" on this point but lacks in other areas. Sphere can support the development of its successor just as it can support its predecessors though its spheres. Finally it is important that Sphere be written so that even average users can once again regain total control of systems that have been walled off by corporate interests or have evolved into unclimbable cliffs of complexity and confusion. By doing this, by reducing the workload of a system operator, so much additional time and talent can be directed once again at improving computers. By making these improvements a new golden age of computer design and innovation can be opened with all the rewards that come with it. Sphere is not ahead of its time. Its long overdue! THE REVOLUTION IS COMING!