Arc at XP Day France

Today's arc presentation at XP Day France -

Introducing Welder

When I read Gödel, Escher, Bach, I decided to drop my literature degree and take a computing one instead to learn Lisp. That didn't happen ... I got as far as expertly and elegantly configuring a key binding or two in my ~/.emacs file while struggling with java-mode. Many years later, arc came out, and Paul Graham being who he is, I just had to take a look. Maybe I could build an online ecommerce solution with this stuff, get rich, and turn into an Angel.

Anyway, the first thing I noticed was that I didn't have an editor for arc. Intellij wasn't going to cut the mustard, I wasn't falling in love with Textmate despite those gorgeous RoR screencasts, and I didn't feel like learning DrScheme. And none of these options would give me arc-aware highlighting and evaluation in any case.

So I decided to try what anyone learning a new language should try: write an editor for it. In it. On top of that, I needed something to drive rainbow that would demonstrate easy java integration without polluting the elegance of the arc spirit. And lo, Welder was born. Welder is an arc editor, written in arc. You'll have to forgive, or at least tolerate, the pun. The name "arc" invites it, what else could I do?

Don't get excited - Welder isn't quite ready for prime-time. It's too slow to parse a file (around 5 seconds for arc.arc - 2.5k lines, 13k tokens). The slowness is in the tokeniser, so I'm concentrating my speed-up efforts there. For smaller files (~3k tokens), its performance is tolerable.

These were the minimal required features I started with:

• Syntax colouring: strings, numbers, characters, should have their own colours, and symbols should be coloured differently depending on their current bound value, if any. This way it's easy to see if a local variable is shadowing a globally bound symbol, and what kind of a thing it's shadowing.


• Paren-matching - highlight matching parentheses, brackets, string delimiters, and any other kind of delimiter that might arise in the future. Also, highlight unbalanced delimiters.


• Easy selection of forms: I'm addicted to intellij's "widen selection" - I want to be able to select the entire form enclosing the current expression, in such a way that repeated application selects an increasingly wider scope. This makes copy/paste operations a lot simpler.


• Eval: I want to eval the currently selected expression, or the whole file, with a single keystroke


• Htmlify: there's no point in writing arc if you're not going to blog about it. But what a pain to reproduce all that syntax highlighting! and besides, if arc gets its herokugarden, this might be a useful part of it.


• Configurable keystrokes for all actions


• Pretty-printing (doesn't work terribly well)


• As well as all the usual editor facilities: undo/redo, cut/copy/paste. Fortunately, java/swing comes with this kind of thing pre-packaged.

Added later:

• Font-size control - essential to demo welder on a projector


• "pop-form" - replace (foo (bar)) with (bar) if the cursor is on bar


• "push-form" - replace x with ( x) - these two together are really useful for inserting/removing a little do, atomic, time, debug or whatever you need for quick refactorings

Note the paren-matching:

And easy form-selection:

Context help for symbol under caret (help text, if present, and the help system, are entirely thanks to the tireless efforts of anarki contributors)

And the complete list of current keybindings:

Welder is still outrageously incomplete. Here are the kinds of thing I would like to add:

• Visual macro expander


• Common refactorings that are annoying to do by hand - toggle between let and with for example


• Navigate to source of symbol declarations


• Appropriate highlighting for lexically-bound symbols (I don't know how to do this when the scope is created by a macro, other than macexing everything and working backwards)


• Runtime analysis


• and everything else ...

Please suggest and vote for your favourite features!

Welder requires anarki, and is distributed with rainbow on github at http://github.com/conanite/rainbow/tree/master.

calling java from arc

Rainbow adds six builtin functions for calling java code from arc. Theoretically this is all you need to interact with any java library from arc code.

java-new (classname . args)

Create and return a new instance of classname, passing args to the constructor

java-class (classname)

Return a java-object wrapping the class identified by classname

java-invoke (obj method . args)

Invoke method on obj with args. 'defcall binds this function to java-object so that you should only rarely need to call this explicitly. Return the value of the method if there
is any, wrapping into arc types where possible.

java-static-invoke (classname method . args)

Invoke static method on classname with args.

java-static-field (classname fieldname)

Return the value of the specified static field

java-implement (interfacename strictly methods)

Returns a java-object which implements the java interface identified by interfacename, the implementation being provided by the arc code given in methods, where methods is a hash mapping method-name to corresponding implementation. The strictly parameter determines whether an error should be raised if methods does not include all of the methods defined on the interface. java-implement is necessary for implementing, for example, mouse and keyboards handlers for a desktop application, or servlets and filters for a web application.

Following in the arc tradition of using arc-code wrappers for builtin functions (eg 'w/stdout for 'call-w/stdout, 'atomic for 'atomic-invoke, 'thread for 'new-thread), rainbow provides some wrappers to make java interaction a little less ugly and possibly a little more arcish. The 'implement macro removes some boilerplate around 'java-implement, so you can create an implementation of, for example, an iterator (where xs is a list) thus:

(def to-iterator (xs)
  (implement "java.util.Iterator"
    hasNext (fn () xs)
    next    (fn () (pop xs))))

The 'bean function instantiates and configures a java object via its setter methods. Here is an example of creating a JFrame object, specifying its bounds and title:

(def frame (left top width height title)
  (bean "javax.swing.JFrame"
    'bounds       (list left top width height)
    'title        title
    'contentPane  (box 'vertical)))

Behind the scenes, bean is going to call setBounds, setTitle, and setContentPane on a newly-instantiated JFrame object, finally returning the JFrame.

There are lots more goodies defined in rainbow's swing.arc, but that's a topic for another day.

Let There Be Local Variables

Actually, there aren't any. There are only function parameters. But there's a workaround: use functions.

Suppose you're writing a token analyser for some input. 'token-generator is a generator function. You need to determine the type of a token - return 'string if you think it's a string, 'char, 'int, etc. Naïvely, you might

(def token-type (token-generator)
  (get-token-type (token-generator)))

(def get-token-type (tok)
  (if (is (tok 0 #\")) 'string
      (is (tok 0 #\#) 'char)
      (is-digit (tok 0)) 'int))

So you get your input from 'token-generator, but you're really interested in 'tok. So the function 'get-token-type provides the local variable 'tok for you to work with. The disadvantage is that your namespace gets polluted with a zillion little functions that are only useful to their immediate neighbours. Wouldn't it be nice to, like, click Refactor > Inline Method like you can in a real language?

Well, you can.

(def token-type (token-generator)
  ((fn (tok)
    (if (is (tok 0 #\")) 'string
        (is (tok 0 #\#) 'char)
        (is-digit (tok 0) 'int))) (token-generator)))

The ( (fn (tok) (if blah)) (token-generator) ) is a call to an anonymous function that declares the 'tok parameter for you to work with.

Now your namespace is no longer polluted, but you have this damned ugly thing in the middle of your function. No ordinary human is expected to read this.

Enter 'let

'let is a macro that transforms its humanly-readable input into something resembling the mess above.

(def token-type (token-generator)
  (let tok (token-generator)
    (if (is (tok 0 #\")) 'string
        (is (tok 0 #\#) 'char)
        (is-digit (tok 0) 'int))))

'let lets you pretend you are actually declaring a local variable. Coming from another language, you might think that this is, really and truly, a local variable. It's not. Behind the scenes, it's an anonymous function creating a lexically scoped 'tok parameter. But you don't care. You have your local variable.

Here is the pattern:

(let a b stuff)

becomes:

((fn (a) stuff) b)

'with is a more general purpose macro that does the same thing, but for multiple variables.

(with (a 1 b 2 c 3) stuff)

becomes:

((fn (a b c) stuff) 1 2 3)

'let exists because in the special case of binding just one variable, we don't need parens to separate the variable bindings from the body of the code. Just because you write in lisp doesn't mean you actually like parens ...

To Do: If Only ...

One of the simplest and most pervasive macros in arc is "do". "do" is useful when you need to evaluate a sequence of expressions, but you only have room for one. A common case is when evaluating conditionals:

(if a b
    c d
    e)

This corresponds to the following in java:

if (a) { 
  return b; 
} else if (c) {
  return d;
} else {
  return e;
}

In java, we can execute as many statements as we desire within a { } block; arc however allows only a single expression as return value. So if we wanted

if (a) { 
  b();
  c();
  return d; 
} else if (e) {
  return f;
} else {
  return g;
}

we can't just write

(if a b c d
    e f
    g)

Hence, do.

(if a (do b c d)
    e f
    g)

do expands into a call to a no-arg function containing b c d -

((fn () b c d))

- it defines an anonymous zero-arg function and immediately calls it. Hence, b c d are grouped in a single expression, and will work as desired inside a conditional:

(if a ((fn () b c d))
    e f
    g)

arc functions

So the long and the short of it is, we get arc to evaluate expressions. Atoms mostly evaluate to themselves: strings, characters, and numbers.

arc> "foo"
"foo"
arc> 12
12

Symbols are also atoms, but have the special property that they may be defined to have some other value. Quoting a symbol means it doesn't get evaluated.

arc> 'bar
bar
arc> (set bar "toto")
"toto"
arc> bar
"toto"

Lists are another story: when arc evaluates a list, it assumes the first item of the list is a function, and the remainder are arguments.

arc> (+ 1 2)
3
arc> (expt 9 2)
81

The first item is not necessarily a function: it might also be a reference to a macro, or in anarki, an object for whose type a function has been prepared via defcall. For another day. The first place in a list is usually referred to as "functional position". So, the procedure for evaluating (a b c d), is to evaluate each of a, b, c, and d (order is unspecified). a should evaluate to a function, which is then called with the values of b, c and d as arguments.

But, the object in functional position doesn't have to be a symbol pointing to a function. It could also be an inline, anonymous function.

arc> ((fn (a b) (+ a b)) 23 67)
90

Disadvantage: it's dead ugly, and totally unreadable. Nest a few of these and you have parenthesis soup. It turns out this is one of the fundamental organising principles of arc code. So why bother?

• group expressions


• declare local variables


• iteration


• all sorts of wonderful things

- all this in a neat and concise way, without namespace clutter or parameter hell. Except, we need macros to make it happen.

Macros: a future post.

Lexical Scope and Closures

Lexical scope is not a radical departure from java: the concept already exists. When we create an inner instance class in java, the inner class has access to all the instance variables of all enclosing classes, and even to local variables if it's an anonymous inner class. Similarly, in arc, a locally-defined function has access to all the variables of all enclosing functions. This set of variables and their values is called the closure of the function.

 (def powerfn (y) (fn (x) (expt x y)))
 
 (set square (powerfn 2)) ; --> square is (fn (x) (expt x y)) with y = 2
 (set cube (powerfn 3)) ; --> cube is (fn (x) (expt x y)) with y = 3

 (square 5) ; --> 25
 (cube 5) ; --> 125

In this example, the variable y retains its value - 3 or 2 - long after powerfn has exited, in the closure of the function returned by powerfn (the function that is assigned to square and cube)

When rainbow finds a locally-defined function in the course of evaluation, it creates a Closure object which encapsulates the function in question, and its closure. The Closure thus created can subsequently be stored in a variable, and invoked anywhere at any time. The implementation isn't optimal however: it keeps the entire closure, rather than just the closed variables that are actually used by the function in question.