Kevin Reid's blog

A machine I used to use to host some web services, bots, and repositories became no longer accessible from the Internet, as a result of which I've had to move what I was serving from it; some to switchb.org, some to personal machines.

I took the opportunity to clean things up a bit, as a result of which I now have better backups, more polished services, and know a little bit more about configuring Apache — though not as much as I perhaps should.

• My Subversion repositories are now served over HTTP, and therefore browsable; and they are now backed up daily (using svnsync triggered by a cron job) to my laptop, and thence to all its backups.

  (I wasted several minutes on remembering that cron will ignore the last line of a crontab file if it doesn't end with a newline; after listening to me grumbling about this, someone made a suggestion to end the file with a comment, so that the last line is harmless whether ignored or not, and also reminds one of the issue.)

  If you have a working copy of one of my repositories (E-on-CL, E-on-JavaScript, MudWalker, Den, etc.), here's a guide to the changed URLs.

• My other Tahoe-LAFS volunteer grid storage node is now residing on a machine on my home LAN.

• Finally, some simple data-querying web services I wrote for Waterpoint's word games have now been moved to switchb.org; I also took the time to prettify their URLs (no cgi-bin or .cgi) and write documentation.

I haven't yet gotten to working on the bots, darcs repositories, or miscellaneous other stuff I had there.

(Pondering moving my blog over to switchb.org as well so as to not have ads, especially now that I found I can still have LJ-friends by way of OpenID. (Hm, but reading friends-locked posts over RSS might not work since there's no username+password for LJ to accept. Anyone have experience with that situation?))

Apache configuration questions:

1. If I have multiple variants of a document (e.g. foo.html foo.pdf foo.txt) handled by MultiViews, so the canonical URL of the document is extensionless (“foo”), how do I properly control the default variant to serve in the event that the client does not express a preference via the Accept header? (Without doing so, I found that it would serve the .txt version, whereas I would prefer the HTML.) All that I found that worked was to create a type map file named “foo” with quality values, and force it to be interpreted as a type map using <FilesMatch>. This seems kludgy to me.
2. What is the right way to serve CGIs, not in a designated cgi-bin directory, and without any .cgi extension in the URL? I initially tried to apply mod_rewrite, but I couldn't get it to work such that /foo internally contacted foo.cgi whereas /foo.cgi redirected to /foo. I resorted to another <FilesMatch> explicitly listing the name of each CGI and doing SetHandler cgi-script.
3. What is the right way to handle “testing” vs. “deployment” configurations, where the relevant Directory, Location, etc may be different depending on which server or subdirectory the site is set up on? I see that one may use environment variables — should I just set up variables containing the path prefixes for the particular host before including the generic configuration file?

Has anyone invented a common protocol for, in a desktop environment, updating status (available/away/do-not-disturb and “what I'm doing” and so on) in social applications? I have 5 different such applications (iChat, Colloquy, Skype, MudWalker, Steam), and I usually don't bother to update all of them since it's too much trouble to do regularly. Is there some protocol that, say, the Linux-on-the-desktop folks have invented for having just one widget to inform everything?

[edit] Clarification: I mention Linux-on-the-desktop just because I don't know much aboutit and there seems to be some innovation from that direction in this kind of area. The desktop in question is Mac OS X; I'm expecting to have to write the software myself; I just want to not reinvent the wheel with regard to protocol.

There seems to be a recurring misconception about writing .asd files. It is not necessary to start your asd with (defpackage :foo.system ...) (in-package :foo.system) to “avoid polluting the package asdf loads your system in”.

Every time an .asd file is loaded, ASDF creates a fresh package to load it in. The relevant code from ASDF is:

(defun make-temporary-package ()
  (flet ((try (counter)
           (ignore-errors
                   (make-package (format nil "ASDF~D" counter)
                                 :use '(:cl :asdf)))))
    (do* ((counter 0 (+ counter 1))
          (package (try counter) (try counter)))
         (package package))))

(defun find-system (...)
  ...
      (let ((package (make-temporary-package)))
        (unwind-protect
             (let ((*package* package))
               ...
               (load on-disk))
          (delete-package package)))
  ...)

So, whenever you load an asd file, the package is fresh and :uses CL and ASDF.

There are reasons to define a package for your asdf system definition.

• If you are creating any classes, such as custom ASDF component types, then, since classes are not garbage collected (MOP specifies that the list of a class's subclasses can be retrieved, and no implementation I know of uses weak references for the purpose), naming them with symbols in a consistent package (rather than the asdf temporary package) ensures that reloading the system definition does not create useless additional copies of the classes.
• If you are defining functions or other named entities in your system definition (which should of course only be done if they are also necessary to load the system) and want to be able to refer to them, such as for debugging, with nice interned symbol names.
• If you want to be able to (load "foo.asd") rather than going through asdf:find-system.

The disadvantages I can think of for defining a package for your system definitions are that the package list becomes more cluttered (only noticeably if for some strange reason you're loading many system definitions and not loading the defined systems), you're using more of the global package namespace (probably not a significant concern if you use a naming scheme like foo and foo.system), and your code is marginally more complex.

So there are good reasons for (and maybe some against) defpackage in your asd, but package hygiene is not one of them.

[Edited 2009-11-26 to be less worded against defpackage after consideration and feedback.]

I was working on some automated document generation, building the process, revising the input document, and checking the results, and got tired enough of typing "make output/foo.txt && open output/foo.txt" and then going back and editing that line when I wanted to check the PDF version etc. that I wrote a tool which I hope will have more general application as well. I called it “maken”:

#!/bin/sh
# Make files and view the results.
make "$@" && open "$@"

Or it could be generalized into being a combinator, running any two commands, rather than just make and open, specified as the first two arguments:

#!/bin/sh
ca="$1"
shift
cb="$1"
shift
"$ca" "$@" && "$cb" "$@"

(Which, for the Haskell folks, should be called &&&, except that that would be annoying to enter in the shell.)

Every time you write “ | ” in shell, that's concurrent programming.

\documentclass{article}
\usepackage{verbatim}
\begin{document}
  \verbatiminput{\jobname.tex}
\end{document}

What do practicing Java programmers call the part of a method definition that isn't the { body }?

__________/ this bit here \___________
public static void main(String[] args) {
  // not this part
}

A quick search turned up the Java Language Specification, Second Edition calling it a MethodHeader in the grammar, but it doesn't use that term in the text and I want to know what term(s) are used in practice, not solely by language lawyers.

(“Signature” is not correct, as that includes only the method name and parameter types, not modifiers and result type.)

Both of the non-internet-based computer-related classes I'm taking this semester are (only offered as) once-per-week night classes, nominally ending at 9:30. So I'm talking with someone after class — — gee, it's bedtime already.

Late update on Caja-CapTP:

Google Summer of Code is over. I passed based on revised goals, but I'm not happy with the state of the code and I did not complete any significant part of the original plan.

Regarding the items mentioned in my last update:

• Write more documentation/comments.
• Commit as much of the work-in-progress as I can.
  • ...including the incomplete CapTPConnection, even though its tests don't pass yet, so that the partial work can be counted.

I committed CapTPConnection, and found and fixed enough infrastructure (whenResolved, CommTable, SwissTable, deSubgraphKit, etc.) bugs that it starts up and can do a basic operation or two. It's not useful for anything, but it's a lot closer to running than it was at the time of my last update.

Also, I removed dependencies on 'window' so in principle it will operate on a non-browser (server) JavaScript implementation. This has not been exercised because there is no browserless driver for the test scripts yet.

I stated that I would continue working on Caja-CapTP past the GSoC work period; however, with the time occupied by schoolwork, I have not had time or brain space to do so yet. I am not going to abandon the project.

Java programming class (yes, I know Java already) today, introducing loops. Task: write a factorial program. I wrote mine to use int arithmetic if possible, floating-point if not. Test the edge cases, of course. It says:

500! ≈ Infinity

I'm glad I used the “approximately equals” sign.

Stuff done:

• Implemented generic literals.
• Implemented cyclic references in Data-E.
• Various cleanup, debugging support, and committing of stuff previously written.
• Wrote a wiki page documenting Caja-CapTP's major components and files.

Stuff to do:

• Write more documentation/comments.
• Commit as much of the work-in-progress as I can.
  • ...including the incomplete CapTPConnection, even though its tests don't pass yet, so that the partial work can be counted.

A while ago I got the idea to write a “my experiences with JavaScript” post; unfortunately, I’ve forgotten what I was going to put in it. However, yesterday someone said approximately “I need to learn JavaScript in 3 hours, what should I know?”, and I thought of quite a few not-immediately-obvious things to point out.

Had a bit of a breakthrough in debugging facilities: realizing that there's no reason I shouldn't modify my local copy of the Cajita runtime to provide more debug information (and it doesn't matter if it subtly breaks security, even, as long as Caja-CapTP is still compatible with the proper version).

Only a week or so left. I've discussed the matter with MarkM, and he pointed out (again) that I should work toward having useful-to-other-people components even if I haven't finished the one I intended to do.

Toward this end, I am going to work on Data-E (which is already part of the Caja-CapTP project) as a standalone serialization utility (for network and disk) as I understand it will be useful for Shakhar.

Todo items:

• Implement generic literals (description) to make it more JavaScript-natural.
• Implement cyclic object graph handling using Ref promise implementation already done.
• Commit all in-progress work as reasonable changesets (even the tests that don't yet pass).
• Document what works, design decisions, etc.
• Continue to work on CapTP itself as time permits.

I intend to continue working on the Caja-CapTP project beyond GSoC; both simply because it is a project of mine, and to mitigate my failure to complete the planned work on schedule. However, for the next few months it will have to compete with my college work rather than my procrastination.

Rosetta Code is a wiki where a variety of programming problems and language features are demonstrated in many languages, allowing language learning and comparison of features and paradigms.

If you'd like to contribute Common Lisp code to the project, I've just completed a classification of tasks in CL (like my prior one for E), so that it's easy to find the kind of problem one wants to work on at the moment.

One thing I find interesting about working on Rosetta Code is that many tasks bring with them some particular perspective — someone's notion of how programming necessarily works — and it can be a challenge to figure out the analogous way, or best way, to do it in your language is, and then explain it.

I'm having severe getting-around-to-actually-doing-the-work problems; I am far behind schedule. I think the problem is framing this as “work” rather than just another of the projects that I've found interesting and tinkered with. (I also blame Team Fortress 2 and Rosetta Code for providing attractive distractions...)

I've ported the “Ref” facility from E; this is necessary as CapTP is built on Ref semantics (promises, proxies, broken refs). I hope to very soon get the actual CapTP connection module up, then (as I wrote before) “define, document and implement a CapTP-over-HTTP protocol”.

(I previously mentioned implementing the Surgeon (serialization tool) but I then remembered that that's actually irrelevant to CapTP.)

Also: working without Internet access removes a whole lot of potential distractions — and one’s access to online documentation. Luckily I had some source code around which provided examples.

Common Lisp provides compile-file whose purposes is to convert a CL source file into an (implementation-defined) format which usually has precompiled code and is designed for faster loading into a lisp system.

compile-file takes the pathname of a textual lisp source file. But what if you want to compile some Lisp code that's not in a file already, perhaps because you translated/compiled it from some not-amenable-to-the-Lisp-reader input syntax, or because it contains unREADable literals? You can use a "universal Lisp file", which I know two ways to create (use whichever you find cleaner):

(cl:in-package :mypackage)
#.mypackage::*program*

(cl:in-package :mypackage)
(macrolet ((it () *program*))
  (it))

Suppose this is in "universal.lisp". Then to use it:

(defvar *program*)

(defun compile-to-file (form output-file)
  (let ((*program* form))
    (compile-file #p"universal.lisp"
                  :output-file output-file)))

This is just a minimal example; you'll also want to appropriately handle the return value from compile-file, provide an appropriate pathname to the universal file, etc. For example, here's an excerpt of the relevant code from E-on-CL, where I have used this technique to compile non-CL sources (emakers) into fasls:

(defparameter +the-asdf-system+ (asdf:find-system :e-on-cl))
(defvar *efasl-program*)
(defvar *efasl-result*)
...
(defun compile-e-to-file (expr output-file fqn-prefix opt-scope)
  ...
  (let* (...
         (*efasl-program*
           `(setf *efasl-result*
                  (lambda (...) ...))))
    (multiple-value-bind (truename warnings-p failure-p)
        (compile-file (merge-pathnames
                        #p"lisp/universal.lisp"
                        (asdf:component-pathname +the-asdf-system+))
                      :output-file output-file
                      :verbose nil
                      :print nil)
      (declare (ignore truename warnings-p))
      (assert (not failure-p) () "Compilation for ~A failed." output-file))))

(defun load-compiled-e (file env)
  (let ((*efasl-result* nil)
        ...)
    (load file :verbose nil :print nil)
    (funcall *efasl-result* env)))

Note that the pathname is computed relative to the ASDF system containing the universal file; also note the use of a variable *efasl-result* to simulate a "return value" from the compiled file, and the use of a lambda to provide a nonempty lexical environment, both of which are features not directly provided by the CL compiled file facility.

I'm not making progress as fast as I would like; mostly due to assorted distractions rather than Doing The Work.

That said, I've gotten both ends of serialization implemented, at least for non-circular cases: the deJSONTreeKit and deSubgraphKit have both builder and recognizer implementations. (For a terminology introduction, see the serialization paper.)

Next up is the surgeon (high-level serialization interface), and designing uncallers suitable for JavaScript. I definitely need to increase my rate of progress to get this done on schedule.

I completed enough of the E-on-JavaScript improvements, and wrote the beginnings of one Data-E kit in Cajita, together with the Updoc tests for it, and a Makefile to cajole the JavaScript and "animate" the Updoc - convert it into a HTML page that actually runs the tests.

I also improved various readme files and pages, hopefully such that someone else can get it all installed on their own system starting from my project page without too much prior knowledge.

Near-term plan: Put aside the EoJS, and get the Data-E kits working; then the surgeon; then the CapTPConnection; then define, document and implement a CapTP-over-HTTP protocol.

I have begun the work on my GSoC project, Caja-CapTP.

I am currently working on improving E-on-JavaScript in order to be able to use its Updoc facility as the test framework for the Caja-CapTP code.

(If you don't know, Updoc is a test system which works by rerunning a recorded or made-up interactive session and confirming that the output is the same as before. The advantage of this system is that you can write test cases very simply as sequential code without gobs of assertFoo() for every particular attribute you think of testing.)

You can see my commits to EoJS at CIA.vc.

I am also learning more about exactly how Cajita works in areas such as library loading; the documentation in this area needs improvement, which I am going to work on myself and/or ask the Caja developers to clarify.

I have been accepted to Google Summer of Code 2009!