Currying in Lispz


Functional programmers love currying for some reason. [Haskell Brooks Curry](https://en.wikipedia.org/wiki/Haskell_Curry) was an American mathematician who lived through the first 80 years of the 20th century. So, _currying_ is named after a person and is not a description – except possibly that it spices up your code :). Currying is the epitome of a higher level function. You supply it with a function that has a fixed list of parameters. It returns a function that can be called with less parameters. Until all the parameters are supplied, the curried function returns another function. You will probably get a better explanation here: https://en.wikipedia.org/wiki/Currying. Better still, an example. In Lispz, _curry_ behaves like _lambda_ while providing the additional functionality…

(ref add-3 (curry [a b c] (+ a b c)))

(add-3 1 2 3) ## ==> 6; is the same as
(((add-3 1) 2) 3) ## is the same as
((add-3 1 2) 3)

This doesn’t look very useful (probably because it isn’t). Here is a simple use of currying where both _first_ and _second_ are functions that return the appropriate entry when given an array.

(global dot (curry [index array] (#join '' '(_res_=' array '[' index '])')))
(global first (dot 0))
(global second (dot 1))

There are also functions for third, fourth and last. It is a way to make your code more declarative, but is nothing more than a convenience. I could have written

(global first (lambda [array] (get array 1)))

You will soon recognise code from functional programmers who favour currying. Their functions will be defined with the parameters in an unusual order so that the parameters likely to change are at the end. If you are lucky enough to have a function with two parameters you can reverse the order. Pull is curried, so we can use it to create a new function that knows about a stateful cache and can retrieve items by name.

(global get-from (flip lispz.globals.dot))

(ref commands {a: 1 b: 2 c: 3})
(ref command (get-from commands))
(command "b")

I do not use currying a lot, preferring explicit definitions, but when it is useful it is very useful. The following example keeps a cache of arrays accessed by name. When a new name is used a new array is created.

(global stateful.cache (curry [store update key]
  (or (get store key) (do (store.update! key (update)) (get store key)))
))
## ... much later
(ref exchange (stateful))
(ref observers (stateful.cache exchange (=> (stateful []))))
## ...
((observers "my-key").push! my-value)
Advertisements

Statefullness and Concurrency


Imagine a stateless conversation…

Mr Stateless — How are you today?
Ms Stateful — Great thanks
Mr Stateless — How are you today?
Ms Stateful — I already told you, Great

… and so on. Not very productive, is it? So, when a functional programmer next tells you that state is evil, act like Mr. Stateless and drive them crazy!

What your functional programming friend is really saying is that for many reason state needs to be contained. Uncontrolled state and stateful objects make software —

  • hard to test and debug because it is difficult to tell what state it is in at any given time.
  • more difficult to reason about as time and order must be accounted for.
  • Makes multi-tasking, multi-threading and multi-core work much more difficult.

I would like to expand on the last point. Java uses the synchronize keyword to protect actions on common data from changing it in the middle of an operation. Firstly the developer must be savvy enough to avoid deadlocks. These are where two or more pieces of code are blocked because they are each waiting on synchronized data held by the other. If synchronization is too fine-grained there will be significant delays due to checking and task switching. If too course grained, the application will behave like there was only one CPU and thread since every other thread is waiting on data locked elsewhere. This is a very common bottleneck in large Java applications.

In short synchronizing is hard. By isolating and quarantining known stateful operations, the code is easier to reason about and test.

Functional languages have a diverse set of features to quarantine stateful operations. Haskell has stateful monads, Erlang has message passing and Scala has Actors to make a few.

Lispz is in a different environment. All code and DOM manipulation (display) in the browser is on a single thread. There is still some multi-tasking since code can switch to a different context when waiting for asynchronous events (http requests, timeouts, user interaction, etc). Unlike many server-based systems, a developer can expect data to remain unchanged until the code makes an asynchronous request. After that all bets are off.

Also, browser based software must deal with the intrinsically stateful DOM (document object model) that lies behind the page content we see.

For these reason Lispz

  • isolates state changes to specific objects and arrays
  • uses a syntax that highlights as dangerous in an editor
  • doesn’t make stateful actions so difficult as to make the system hard to use.

Lispz does not support variables directly. Everything is accessed by reference. As with all of Lispz, this is by custom, not by policing. This means that with a knowledge of JavaScript you can cheat – just don’t do it.

A dictionary can created is a way that it can hold stateful data. It can also be provided with seed data to start it off…

(ref stateful-data (stateful))
(ref stateful-data (stateful) { name: "hello" seed: { more: "less" less: "more" }})

Once a reference to a stateful object has been created it is possible to change data from the root of the dictionary tree using _merge!_, _replace!_ or _delete!_.

(stateful-data.replace! { name: "goodbye" })
## ==> { name: "goodbye" seed: { more: "less" less: "more" }}

Decoupling Web Components in RIOT


Every framework is designed to make our life easier. But, even the name indications the limitations. A framework will make it difficult or impossible to do some tasks. Often this is good, but sometimes very annoying and limiting. As software developers we spend a good portion of our time finding and implementing workarounds or solutions to limitations in our tools and frameworks.

There is a sometimes obscure difference between workarounds and solutions. Typically a workaround is a hack that takes advantage of an edge-case in the framework. By contrast a solution is an add-on or plugin to the framework that provides new functionality.

The particle component is a solution that is best understood by working through the problem.

RIOT is a clean-room implementation with a structure similar to web components (without waiting the years for the standard to be fully adopted). I suspect the problem I am about to describe exists in Polymer, Angular 2 and other systems of web components. These systems will have other solution.

The Problem

Web components are often made up of other web components. Sometimes we need to define content in an outer component for use in an inner. RIOT has the yield tag for this purpose:


<!-- definition -->
<my-tag>Combining "<yield/>" in a RIOT tag</my-tag>
<!-- usage -->
<my-tag>This is <b>HTML</b></my-tag>

RIOT even allows named yields. This is a tightly coupled solution. The problem arises when we want to add to the DOM of an inner component.


<!-- definition -->
<panel>
  <div>
    <div class=panel-heading>filled by higher level parent</div>
  </div>
</panel>
<code-editor>
  <panel></panel>
</code-editor>
<!-- usage -->
<code-editor>
  I want to test to display on the toolbar or the inner panel component
</code-editor>

The Workaround

I thought of this as a solution at the time – until a RIOT update changed processing to disallow nested yields.


<!-- definition -->
<panel>
  <div>
    <div class=panel-heading><yield from=heading/></div>
  </div>
</panel>
<code-editor>
  <panel><yield from=heading><yield to=heading/></yield></panel>
</code-editor>
<!-- usage -->
<code-editor>
  <yield to=heading>
    I want to test to display on the toolbar or the inner panel component
  </yield>
</code-editor>

As you can see I am passing HTML content down through the layers to the destination. It is still closely coupled and it requires a syntax change at the top level to work.

The Solution

In short – use components.


<!-- definition -->
<panel>
  <div>
    <div class=panel-heading>
      <particle class=riot if={ opts.toolbar } name={ opts.toolbar }/>
    </div>
  </div>
</panel>
<code-editor>
  <panel toolbar={ opts.toolbar}></panel>
</code-editor>
<!-- usage -->
<code-editor toolbar=code.toolbar>
  <particle class=riot name=code.toolbar>
    I want to test to display on the toolbar or the inner panel component
  </particle>
</code-editor>

Differences to Yield

* A particle is decoupled – it can be created and located in otherwise unrelated HTML
* A particle is created in the source context. Yield is text based, so it will generate in the target context. This makes <yield> more difficult to use opts – making it even more tightly coupled.
* Particle names/paths are application global.
* It does not matter whether the source or destination tags are processed first. If they are not nested, care must be taken to update both together.
* The particle source is extremely simple – (particle.lispz). If you want a JavaScript version, compile the lists at http://lispz.net.

Does HTML5/JS(2016) === Windows3(1994)


Microsoft Windows prior to Windows’95 had what was called ‘cooperative multi-tasking’. This meant that a program had full-time access to the CPU until it (the program) released control to the operating system. The screen did not update and the mouse and keyboard did not respond if a program went off on a lengthy calculation or an unintentional infinite loop.

It was a necessary approach in the days of the 8086 and 80286 processors. Many complained about unresponsive systems and blamed the OS. In fact, with well written software, poor responsiveness was rare.

Now, wind forward 20+ years and look at the current movement to single page applications and the efforts to allow a browser application to run stand-alone — what I call disconnection tolerant applications. JavaScript is what is called ‘single threaded’, meaning that code has full-time access to the virtual machine inside the browser. We use the term ‘event driven’ over ‘cooperative multi-tasking’. I think the latter is more accurate, since a preemptive system could also have event driven code.

I like the approach. I accept the responsibility for not locking the UI when it gives me freedom from many of the issues in preemptive multi-tasking. Of course it helps that we have much faster computers and a good preemptive kernel to handle I/O.

In Search of Simplification — in Languages


I was originally going to use this blog to discuss dynamically typed languages. Everyone I talk to or read is so committed to static typing that I felt completely alone in my view. At least I did until I read the following blog by Daniel Lebrero:

The broken promise of static typing

There is a lot of discussion about the validity of the data interpretation and not about the results. Peer review in action. Even without further analysis, the experiment alone gave me pause for thought.

Personally I prefer dynamic typing. The data from the article implies that bugs that reach production are little different between dynamic and static typing. If dynamic typing was so evil, then static typing would dominate the top of the graphs, regardless of what normalisation you add. It doesn’t, so I feel happier about my preferences.

How about functional programming? Again the charts show an even distribution between functional and OO paradigms. This implies that the much touted referential integrity does not help as much as I supposed. I am sad about this since I put a lot of effort these days into isolating statefulness. Perhaps I should consider it a hobby rather than a way of improving reliability?

Daniel’s conclusion is that simplicity is the key. Look at the name of this blog. For that reason alone I want his conclusions to be right. Sadly if this were the case, Scala would fare much worse and JavaScript much better. And if it were complexity of ecosystems then Clojure, Scala and Java would not be spread across the spectrum.

I suspect that the spread is due to people, not technology. The three with the worst bug-ratio are languages used commonly in business by programmers who are doing a job. As we go up the list we reach languages that people are passionate about and therefore take more care. Of course this view is as baseless as the others. Why is the bug could for Haskell so high? Probably because passionate people are more likely to want fixes. If so, why is clojure so low?

I give up. There is no way with the limited data to make any inferences. That makes me happy – because I don’t have to feel guilty for preferring dynamic typing – or not thinking that language XXX is the best thing ever.

The ins and outs of Javascript Promises


When I first saw the promises specs and early shims for JavaScript, I could not see the benefits. With careful use of callbacks in closures, the code was less obscure that a chain of .then functions – particularly in CoffeeScript. Yes, there was unified error handling, but that was not enough to encourage change.

In the last year two additional factors encouraged me to change my mind.

  1. Promises are now native to all modern browsers (and node).
  2. Lispz, as a functional language with an excess of parentheses brings back the callback hell of pure JavaScript (over CoffeeScript).

Last week I had a problem where promise rejection deep with a promise stack were going to window.onerror rather than passing up the line. A friend and I spent a couple of hours on understanding promises better. Here are some of our findings.

Basic Promises look like callbacks

var summer = function(accumulator) {
  return new Promise(function(resolve, reject) {
    setTimeout(function(){resolve(accumulator + 1)},1)
  })
}

var show = function(accumulator) {
  console.log(accumulator)
}

summer(0).then(show) // ==> 1

In the example above, using the setTimeout callback would produce less code with only a little loss of clarity..

Promises chain

Life gets more interesting when you need a sequence of asynchronous expressions to occur in order where each event requires information from a previous event. In other words, making asynchronous code synchronous.

summer(0).then(summer).then(summer).then(show) // ==> 3

This is called “callback hell” because each callback is inside the one before causing an unreadable level of indented code.

var summer = function(accumulator, cb) {
  setTimeout(function(){cb(accumulator + 1)},1)
}

var show = function(accumulator) {
  console.log(accumulator)
}

summer(0, function(accumulator) {
  summer(accumulator, function(accumulator) {
    summer(accumulator, function(accumulator) {
      show(accumulator)
    })
  })
})

Now picture the above with code that actually does something. As clear as mud?

Promises can stack as well as chain

If the code passed to a then returns a promise, this is added to the promise chain as you see above. There is nothing stopping the returned promise to be a chain – and so on to any depth. It may not seem much of a difference, but is critical to flow control. In a more complex async interaction, promises from otherwise unconnected functions need to be called in a controlled way.

Below is a complex example in Lispz for reading and compiling files from GitHub. When in Lispz converts to then in JavaScript. By convention I add > to library functions that return a promise.

    (ref lispz-js    (repo.read> "lispz.js"))
    (ref groups      (when repo.entries> (group @)))
    (ref modules     (when groups  (build-modules @.modules)))
    (ref riots       (when groups  (build-riots @.riots)))

    (when (promise.all modules lispz-js riots) [sources]
      (ref  code  (list.flatten [["window.lispz_modules={}\n" sources]]))
      (return (repo.write> "ext/lispz.js" (code.join "") "lispz release code")
      )
    )

This may be a hard read for the non-lisp-aware. Groups returns a promise that when fulfilled give a list of files in GitHub. When this promise fulfills the build functions are called to read the files and store them. Promise.all is a JavaScript Promises function that resolves when all in the list resolve. In this case one everything is read the target file can be written.

Then and Catch always return a promise

It doesn’t matter what you return from a then clause – it will always be wrapped in a promise. On consideration, this is obvious. Since then is only called once a promise is fulfilled it only has the option of returning a promise.

summer(0).then(summer).then(function(){ return 23 }).then(show) // ==> 23

This is useful for caching. The first call is asynchronous – loading data from the server. All later calls read from the cache and can return the results immediately.

function load(file_name) {
  if (cache[file_name]) return cache[file_name]
  return new Promise(function(resolve, reject) {
    load_from_server(file_name, function(err, result) {
      if (err) return reject(err)
      resolve(result)
    })
  })
}

Exceptions in a promise turn into rejection

The reject function is tightly coupled to the promise. In practice the asynchronous action may be levels need in the call stack. Do we need to pass the reject function down to everyone who may need it. This is not viable if you are using libraries you did not write. Fortunately the designers of Promise-A+ accounted for this. Inside a promise an exception thrown is limited to the promise context and is converted into a call to reject. So, throw away.

There is a caveat. If you have a call-back inside a promise, any exceptions thrown inside the callback is in a different context. They are not converted to promise rejections. This is the only time you will need try and catch.

 

Expressions vs Statements


A statement is made up of an expression and a terminator. It does not return a value. It’s only way to effect the rest of the program is by side-effects.

Expressions in computing are related to those in mathematics. Statements were a historical necessity. In the beginning computers were very expensive while still only able to do one task at a time. By the time FORTRAN was first in use in 1958, the best way to share the resource was by presenting a program as a stack of punched cards, one per line of code. A standard emerged where each card represented 80 characters. The first 72 were read by the computer. The last 6  could optionally hold a statement number that was ignored by the computer reader. There were large mechanical sorters so that if you dropped these sometimes very large decks of cards then they could be sorted back in order. Programmers were in the habit of numbering cards 10 apart so that new cards could be inserted as the program needed changing. For purely practical reasons each card needed to stand alone so program changes could be effected without re-punching the whole deck.

From another perspective, assembly language representations of computer code are also statements, so perhaps the concept was meant to be.

Every computer language meant for people to read and write need a means of abstraction. With statements we can do that with variables.

var pi = 3.14159;
var radius = 132;
var area = pi * radius * 2;

In Lispz, the expressions can look almost exactly the same.

(ref pi 3.14159)
(ref radius 132)
(ref area (* pi radius 2))

… and I would probably write it that way for clarity, but given that each is an expression, I could also say

(ref area (* (ref pi 3.14159) (ref radius 132) 2))

Under the covers, ref translates to var as Lispz compiles to JavaScript. The are conceptually different – ref is short for reference and implies that it points to the expression rather than being a container for the evaluation result. This is more poignant in languages like Haskell that have lazy evaluation. References in these languages actually point to the expression that is only evaluated the first time it is used. This is like promises in less radical languages.

Another interesting fact. Have you noticed that some languages, like CoffeeScript and Ruby will return the last expression without the need for a return statement? This comes from functional statement-less languages such as Haskell where the only value a function can return is the last (and in the case of Haskell only) expression.

A clean functional programming function (say that three times quickly) will have references for clarity and one expression. Haskell enforces this, while others such as Lispz leave this to the programmer.