Hazelnut: dynamically create a kernel in areflexive language

Benjamin Van Ryseghem

20th May 2011

Contents

1 Hazelnut 31.1 Kernel Class Identification . . . . . . . . . . . . . . . . . . . . . . 31.2 Kernel Isolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.2.1 References to unwanted classes . . . . . . . . . . . . . . . 71.2.2 Reroute dependencies from original classes to Hazel classes 9

1.3 Image Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Chapter 1

Hazelnut

Hazelnut is one of the Seed project, where Seed was originally composed bydifferent projects whose goal is to generate new kernels, all based on the pro-ject Micro-Squeak (Squeak is another implementation of Smalltalk) For now weessentially distinguish two of them.

• PineKernel: it is a port of Micro Squeak in Pharo

• Hazelnut: it is the building of a kernel in Pharo starting from the Pharokernel.

This project is composed by three different parts, the kernel collection, thekernel isolation and finally the image creation.

1.1 Kernel Class IdentificationGoal: The goal of this part is to create an alternative SystemDictionary (aSystemDictionary is a namespace holding all the classes of the system) startingfrom the Pharo one and to collect classes which are needed to build the kernel.

Problems:

• Which classes need to be collected ?

• How do we fill up the new SystemDictionary with those classes ?

Solutions:

• A first naive approach is to collect every classes Object depends on inorder to have an autonomous system. But due to bad dependencies in thesystem, the kernel collected this way contains half of the Pharo classes.This is clearly not working. Therefore we have decided to have anotherapproach. The second and final approach is to provide to the builder the

4 CHAPTER 1. HAZELNUT

list of classes the user wants in the new kernel plus some classes absolutelyneeded by the system1. The problem is that we had to determine whichclasses are the absolutely needed ones. In order to answer this question, atool to analyze classes dependencies has been written and recursively usedstarting from the Kernel package until we had a quite autonomous kernelcomposed of around 200 classes2. This tool also flags bad dependencies,but this part will be exposed in the next chapter (page 9).

• MicroSqueak’s solution to fill up the new SystemDictionary is to recompileneeded classes with a prefix and then to collect them. It’s quite efficientwhen you have 20 classes to copy, but here we have the constraints thatwe do not know by advance what we will copy and then we want to be asfast as possible.

The solution we adopted is to create a new instance of SystemDictionary andto directly copy classes into it without recompiling them. The classes are stillpointing to their original namespace as shown by Figure 1.1.

A

B

C

Dglobals

references

Hazel

#A

#B

#C

#D

classes

B

Dglobals

#B

#D

classes

Pharo

inheritsSystemDictionary creation

Figure 1.1: Step 1 - Copy the classes B and D into the new SystemDictionary

The second step is to make sure that the class and metaclass hierarchy ismaintained in both the environments and that the methodDictionary3 is alsocopied. To be sure to reconstruct the hierarchy, the copy method recursivelyrebuild class, metaclass and superclass hierarchy (see Figure 1.2).

1as Object, ProtoObject or MethodContext for example2Pharo contains around 1800 classes3a methodDictionary is a dictionary implemented in each class and containing all the

methods of the class

1.1. KERNEL CLASS IDENTIFICATION 5

A class

B class

A

B

class inherits

Figure 1.2: Class and MetaClass Hierarchy

Here is the pseudo code in Smalltalk that add a class in the Hazel System-Dictionary and check the hierarchy:

HazelKernelBuilder>>#addAClassInDictionary: class"Add a copy of the class in the Hazel SystemDictionary then answer the copy"

| hazel copy className |className := class name asSymbol.

"Check if the class is already in the dictionary"(self list includesKey: class name)

ifFalse: [^ nil].

hazel := Smalltalk at: #HazelSmalltalk.(hazel globals includesKey: className)

ifTrue: [^hazel at: className].

"If not, add a copy in the dictionary"copy := self copyClass: class.self registerClass: copy.

"then check the superclass"

6 CHAPTER 1. HAZELNUT

copy superclass ifNotNilDo: [:superclass || superCopy |"add the superclass"superCopy := self addAClassInDictionary: superclass."change the superclass"copy superclass: superCopy."then change the metaclass’s superclass"copy class superclass: (superCopy class)].

"Check all literals of all methods"self checkMethods: copy.

"Check all class var"self checkClassVar: copy.

^ copy

The last instructions will be commented in the next section.

The only wrong inheritance which remains is that ProtoObject in the Hazelworld inherits from nil which is still in the Pharo world. But this will be fixedwhen we will change nil (see paragraph 1.3 page 10).

In a nutshell: We are now able to copy wanted classes and needed classes intoa new SystemDictionary, with a good hierarchy, but Hazel classes keep referencesto Pharo ones (see Figure 1.1).

1.2. KERNEL ISOLATION 7

1.2 Kernel IsolationNow that the kernel is created, we need to isolate it by removing dependenciesto the Pharo world. There is two different types of dependencies which need tobe fixed.

1.2.1 References to unwanted classesGoal: Here we want to remove dependencies from Hazel classes to Pharoclasses we haven’t copied.

Problems:

• How to detect unwanted references ?

• How to remove those dependencies ?

• How to be sure it will not crash the system ?

Solutions:

• There are two places where unwanted references can be found:

– In a method: in a method literal there are references to invoke classes(see page ??). Due to that, we can found references to unwantedclasses;

– In a class variable4: we can have an instance of an unwanted class orjust an unwanted class itself.

The solution adopted is to check its methodDictionary and class variablesduring class copy. If unwanted references are found, the following solutionis applied.

• This is the key point of this cleaning step.For class variables, the solution was to set them to nil (in the minimalkernel creation process, only one class variable had to be set this way(HaloFont from StandardFonts).For methods we have considered several solutions. Our first thought wasto remove the method and to recursively remove the sender of the method.But due to class structure, we removed that way almost all methods of thekernel. Then we have though to create a NullPattern object implementingall the methods removed. The problem was to find which kind of answeris expected from each method, and how to dynamically replace the senderin the code source. Finally we chose to remove the method and to keepthe senders.

4a class variable is a variable shared by all the instances of a class

8 CHAPTER 1. HAZELNUT

• We haven’t found a solution to this question. As long as your system canbe changed, you can’t certify that your kernel is totally functional. Weare working on a better isolation of the Pharo kernel to reduce as muchas possible the number of references (see Section ?? page ?? ).

In a nutshell: We have removed the references to unwanted class (see Fig-ure 1.3), but because of that the integrity of the kernel may be corrupted.

A

B

C

Dglobals

Hazel

#A

#B

#C

#D

classes

B

Dglobals

#B

#D

classes

Pharo

references inheritsSystemDictionary creation

Figure 1.3: Step 2 - Remove references to unwanted classes

1.2. KERNEL ISOLATION 9

1.2.2 Reroute dependencies from original classes to Hazelclasses

Goal: Here we want to reroute dependencies from Pharo classes to Hazelclasses to have only intern references in the Hazel kernel. Those referencesare stored into methods literals.

Problems:

• How to change those references ?

Solutions:

• This part is quite simple because the field was well prepared. Only meth-ods refering copied classes are not fixed yet. So for those classes, themethodDictionary had just to be parsed in order to fix literals. And forfixing literals, we just change the Pharo associations to their correspondingHazel one (and we are sure it exists).

In a nutshell: We now have a kernel isolated with only internal references(see Figure 1.4).

A

B

C

Dglobals

Hazel

#A

#B

#C

#D

classes

B

Dglobals

#B

#D

classes

Pharo

references inheritsSystemDictionary creation

Figure 1.4: Step 3 - Reroute the remaining dependences

10 CHAPTER 1. HAZELNUT

1.3 Image CreationGoal: The goal of this part is starting from an isolated kernel succeed to builda new image using this kernel. An image is a snapshot of living objects binarysaved in a file. They basically contains classes and some living instances.

Problems:

• Which technique should we use to create the image ?

• Is it necessary to have a specific Virtual Machine to build image ?

• Is it necessary to have a specific Virtual Machine to read the image ?

• How to successfully replace the Special Objects Array ?

Solutions:

• Two solutions have been tested to create the image:

– The Micro Squeak solution which consists in the collection then theserialization of all the needed objects into a new image. This tech-nique works for Micro Squeak thanks to the limited amount of classesand of objects. Moreover two passes are done on objects with dif-ferent algorithms and due to that difference, we had some missingobjects. In a first time, we tried to fix or rewrite methods, but wefinally decided to consider another solution, dynamically switch theSpecial Objects Array (see Figure 1.5);

– The second solution is to take a lively image and to dynamicallyswitch the Special Objects Array in order to make the unneeded ob-ject garbage collected (see Garbage Collector page ??). The difficultyof this method is that we are drastically modifying the image dur-ing its own execution. Some objects can easily be changed (as nil orCharacter) when some others freeze the Virtual Machine (as String orSemaphore). We think it’s due to the fact that during the executionof the switching method, we are modifying the method context, andthe Virtual Machine points to unaccessible pointers and we got anerror5 (see Figure 1.6).

• In order to avoid the previous problem, we have started to implement newprimitives in C for the Virtual Machine, which force users to use a specificVirtual Machine for image creation.

• The new primitives are only used to switch objects (basically change point-ers to those objects), they should not be needed to run the image. Thatway, users of the new image should not have to use a specific VirtualMachine.

5segmentation fault

1.3. IMAGE CREATION 11

PharoHazelPharo

SerializationHazel

Hazel.image

Figure 1.5: Micro Squeak - Serialization of needed objects

• To replace the Special Objects Array, the first approach was to take thecurrent Special Objects Array, which is a Dictionary, and to replace itsvalues. The problem is that some values called every time by the imageare buffered in the Virtual Machine (those values are nil,true and false)and updated at the opening of the image. So we have decided to use theprimitives become: and becomeForward: which basically switch referencesbetween the receiver and the argument.

In a nutshell: Due to some objects that can’t easily be switched, now we arenot able to generate a new image starting from our Hazel kernel.

12 CHAPTER 1. HAZELNUT

HazelPharo

GC

Hazel

Hazel

Pharo

GC

Figure 1.6: Hazel - Garbage Collection of unneeded objects