Developer Meeting: More on scripting Kate

We are 10 people here at the developer meeting in Berlin. Kate, KDevelop as well as Okteta see a lot of commits. I'll mainly talk about what's happening in the holy Kate land, though :-)
Yesterday I closed a bug requesting an "unwrap" feature in Kate that works like "Tools > Join Lines" but maintains paragraph separation, i.e., empty lines are not removed. This feature is implemented now in javascript. Further infos:

• the bug report
• the javascript implementation

To run the script simply switch to the command line (F7) and write "unwrap". If you have further ideas about useful scripts, don't hesitate to start hacking right away, see also

• how to extend Kate with scripting

Fixes with regard to the scripting support in the last days are

• make reload-scripts work properly
• fix command line completion for new scripts without restarting kate
  
• support quoting in arguments for scripts
• fix crash for invalid commands in scripts

Those fixes will be in KDE 4.4.1. More to come in other blog entries :-)

Scripting Kate

In my last blog I explained Kate's scripting features in KDE 4.4. To better understand how scripting can be used let's look at some use cases.

• join lines: This feature request wants the action "join lines" to not join different paragraphs, i.e. not remove empty lines. We have not implemented this wish, as there are probably users who prefer the current behaviour. This request can be fixed by writing a small script that joins the lines according to the user's wishes.


• reformat paragraph: An intelligent reformatter for paragraphs. Should be rather straight forward to implement.


• XML tools: In KDE3, Kate once had a xmltools plugin. Unfortunately noone ported it to KDE4. The plugin provided lots of very useful features for xml editing. For example, you could select text and then wrap it with xml elements, e.g. "text" would become "<para>text</para>". This is a perfect example for a command line script as well. Any volunteers? :)

Scripting also brings us closer to fixing the following reports:

• macro system: Kate still does not have a macro system. A macro can be interpreted as a group of scripts, executed in a sequence (more or less). The vi input mode already supports pretty complex commands, and the code for scripting is all there. It's just a matter of putting this together so it's usable for users.


• word count: maybe the word count features can be implemented by a script (too slow?). Problem is, that a script cannot show dialogs etc.

To make scripting an even better experience, we still need to implement binding shortcuts to scripts. Again: any volunteers? :)

Extending Kate with Scripts

As mentioned in one of my last blogs, there has been quite some work for JavaScript scripting support in trunk recently. So what will Kate 3.4 (KDE 4.4) will bring? This is explained in detail in the sections:

• Indentation Scripting
  
• Command Line Scripting
• Some Remarks
  

The scripting allows to extend Kate with lots of little helper functions. All users can add scripts as they wish. The documentation here is copied from the official Kate handbook. So instead of using this blog as reference, please use the handbook later for an up-to-date version. To script something useful, you'll need to know the scripting API. All the available functions are documented in the section Scripting API in the handbook as well. Enough said, here we go:

Extending Kate with Scripts

Since Kate 3.4 in KDE 4.4 the Kate editor component is easily extensible by writing scripts. The scripting language is ECMAScript (widely known as JavaScript). Kate supports two kinds of scripts: indentation and command line scripts.

Indentation Scripts

Indentation scripts - also referred as indenters - automatically indent the source code while typing text. As example, after hitting the return-key code the indentation level often increases.

The following sections describe step by step how to create the skeleton for a simple indenter. As first step, create a new *.js file called e.g. javascript.js in the local home folder $KDEHOME/share/apps/katepart/script.

The Indentation Script Header

The header of the file javascript.js is embedded in a comment and is of the following form

/* kate-script
* name: JavaScript
* author: Example Name 
* license: BSD
* revision: 1
* kate-version: 3.4
* type: indentation
* required-syntax-style: javascript
* indent-languages: javascript
* priority: 0
*
* A line without colon ':' stops header parsing. That is, you can add optional
* text here such as a detailed license.
*/

Each entry is explained in detail now:

• kate-script [required]: This text string has to appear in the first line of the *.js file, otherwise Kate skips the script.
• name [required]: This is the indenter name that appears in the menu Tools->Indentation and in the configuration dialog.
• author [optional]: The author's name and contact information.
• license [optional]: Short form of the license, such as BSD or LGPLv3.
• revision [required]: The revision of the script. This number should be increased whenever the script is modified.
• kate-version [required]: Minimal required Kate version.
• type [required]: The type must be “indentation”, otherwise Kate skips this script.
• required-syntax-style [optional]: Comma separated list of required syntax highlighting styles. This is important for indenters that rely on specific highlight information in the document. If a required syntax style is specified, the indenter is available only when the appropriate highlighter is active. This prevents “undefined behavior” caused by using the indenter without the expected highlighting schema. For instance, the Ruby indenter makes use of this in the files ruby.js and ruby.xml.
• indent-languages [optional]: Comma separated list of syntax styles the indenter can indent correctly, e.g.: c++, java.
• priority [optional]: If several indenters are suited for a certain highlighted file, the priority decides which indenter is chosen as default indenter.

Kate reads all pairs of the form “key:value” until it cannot fine a colon anymore. This implies that the header can contain arbitrary text such as a license as shown in the example.

The Indenter Source Code

Having specified the header this section explains how the indentation scripting itself works. The basic skeleton of the body looks like this:

triggerCharacters = "{}/:;";
function indent(line, indentWidth, ch)
{
  // called for each newline (ch == '\n') and all characters specified in
  // the global variable triggerCharacters. When calling Tools->Align
  // the variable ch is empty, i.e. ch == ''.
  //
  // see also: Scripting API
  return -2;
}

The function indent() has three parameters:

• line: the line that has to be indented
• indentWidth: the indentation width in amount of spaces
• ch: either a newline character (ch == '\n'), the trigger character specified in triggerCharacters or empty if the user invoked the action Tools->Align.

The return value of the indent() function specifies how the line will be indented. If the return value is a simple integer number, it is interpreted as follows:

• return value -2: do nothing
• return value -1: keep indentation (searches for previous non-blank line)
• return value 0: numbers >= 0 specify the indentation depth in spaces

Alternatively, an array of two elements can be returned:

• return [ indent, align ];

In this case, the first element is the indentation depth like above with the same meaning of the special values. However, the second element is an absolute value representing a column for “alignment”. If this value is higher than the indent value, the difference represents a number of spaces to be added after the indentation of the first parameter. Otherwise, the second number is ignored. Using tabs and spaces for indentation is often referred to as “mixed mode”.

Consider the following example: Assume using tabs to indent, and tab width is set to 4. Here, <tab> represents a tab and '.' a space:

1: <tab><tab>foobar("hello",
2: <tab><tab>......."world");

When indenting line 2, the indent() function returns [8, 15]. As result, two tabs are inserted to indent to column 8, and 7 spaces are added to align the second parameter under the first, so that it stays aligned if the file is viewed with a different tab width.

A default KDE installation ships Kate with several indenters. The corresponding JavaScript source code can be found in $KDRDIR/share/apps/katepart/script.

Developing an indenter requires to reload the scripts to see whether the changes behave appropriately. Instead of restarting the application, simply switch to the command line and invoke the command reload-scripts.

If you develop useful scripts please consider contributing to the Kate Project by contacting the mailing list.

Command Line Scripts

As it is hard to satisfy everyone's needs, Kate supports little helper tools for quick text manipulation through the built-in command line. For instance, the command sort is implemented as script. This section explains how to create *.js files to extend Kate with arbitrary helper scripts.

Command line scripts are located in the save folder as indentation scripts. So as first step, create a new *.js file called myutils.js in the local home folder $KDEHOME/share/apps/katepart/script.

The Command Line Script Header

The header of each command line script is embedded in a comment and is of the following form

/* kate-script
* author: Example Name 
* license: BSD
* revision: 1
* kate-version: 3.4
* type: commands
* functions: sort, format-paragraph
*
* A line without colon ':' stops header parsing. That is, you can add optional
* text here such as a detailed license.
*/

Each entry is explained in detail now:

• kate-script [required]: This text string has to appear in the first line of the *.js file, otherwise Kate skips the script.
• author [optional]: The author's name and contact information.
• license [optional]: Short form of the license, such as BSD or LGPLv3.
• revision [required]: The revision of the script. This number should be increased whenever the script is modified.
• kate-version [required]: Minimal required Kate version.
• type [required]: The type must be 'commands', otherwise Kate skips this script.
• functions [required]: Comma separated list of commands in the script.

Kate reads all pairs of the form “key:value” until it cannot fine a colon anymore. This implies that the header can contain arbitrary text such as a license as shown in the example. The value of the key functions is a comma separated list of command line commands. This means a single script contains an arbitrary amount of command line commands. Each function is available through Kate's built-in command line.

The Script Source Code

All functions specified in the header have to be implemented in the script. For instance, the script file from the example above needs to implement the two functions sort and format-paragraph. All functions have the following syntax:

function (arg1, arg2, ...)
{
  // ... implementation, see also: Scripting API
}

Arguments in the command line are passed to the function as arg1, arg2, etc. In order to provide documentation for each command, simply implement the 'help' function as follows:

function help(cmd)
{
  if (cmd == "sort") {
      return "Sort the selected text.";
  } else if (cmd == "...") {
      // ...
  }
}

Executing help sort in the command line then calls this help function with the argument cmd set to the given command, i.e. cmd == "sort". Kate then presents the returned text as documentation to the user.

Developing a command line script requires to reload the scripts to see whether the changes behave appropriately. Instead of restarting the application, simply switch to the command line and invoke the command reload-scripts.

If you develop useful scripts please consider contributing to the Kate Project by contacting the mailing list.

Final Remarks

Right now, it's not possible to assign shortcuts to command line commands. Thus, there is no way of executing scripted commands with shortcuts. Volunteers wanted!

The command line scripting can be accessed for all KTextEditor users through the KTextEditor::CommandInterface. That is, you can query a specific command and execute it with arbitrary parameters (The parameter cmd contains the command itself including all arguments. Example: cmd = "goto 65").

Kate's command line itself is actually a quite powerful tool. It's a little bit sad that it's rather unknown. If you want to know more, just invoke "View -> Swith to command line" (shortcut: F7) and start typing text. More details are in the Kate handbook as well.

The Kate scripting API can be found here.

Extending Kate by Scripts

We have seen how scripting basically works for indentation. It's also possible to register commandline functions (The command line is bound to F7 by default, or invoke View > Switch to Command Line). We will consider a small example again: sort the selected text.

/* kate-script
 * name: unused
 * author: foo bar
 * license: LGPL
 * version: 1
 * kate-version: 3.0
 * functions: sorter
 */

function sorter ()
{
    if (view.hasSelection()) {
        var start = view.startOfSelection().line;
        var end = view.endOfSelection().line;

        var text = document.textRange(start, 0, end, document.lineLength(end));

        var lines = text.split("\n");
        lines.sort();
        text = lines.join("\n");

        view.clearSelection();

        document.editBegin();
        document.removeText(start, 0, end, document.lineLength(end));
        document.insertText(start, 0, text);
        document.editEnd();
    }
}

The header line functions: sorter makes Kate Part aware of the function in the script. A list of functions is supported, separated by white spaces. You can use the function by typing 'sorter' in the commandline.
Some todo items:

• provide better JavaScript API. For example: document.textRange() takes 4 parameters. It would be more elegant to take one range or two cursors, just like we do in the KTextEditor interfaces in kdelibs/interfaces/ktexteditor
  
• make is possible to bind scripts to shortcuts. This could be done by e.g. binding commandline functions to shortcuts or implementing a vim-like command-mode in Kate's commandline. How to configure the shortcuts is unclear, though.
• then, think about replacing the C++ implementations of 'uppercase', 'lowercase', 'capitalize' etc. with scripts
• things I forgot...

If you are interested subscribe to kwrite-devel@kde.org and contribute :) We also need indentation scripts, of course!

Kate: More on Indentation Scripting

My last blog was about the theory of how indentation works by using javascripts. Now we will look at a concrete example: a LISP-style indenter. (LISP indentation is easy, that's why it's a good example).
The rules:

• comments starting with ;;; always have indentation 0
• comments starting with ;; should be aligned with the next line
• comments starting with ; should only appear behind code, so they are simply ignored
• every '(' indents and every ')' unindents

lisp.js looks like this:

/** kate-script
 * name: LISP
 * license: LGPL
 * author: foo bar
 * version: 1
 * kate-version: 3.0
 * type: indentation
 */
// indent should be called when ; is pressed
triggerCharacters = ";";
function indent(line, indentWidth, ch)
{
    // special rules: ;;; -> indent 0
    //                ;;  -> align with next line, if possible
    //                ;   -> usually on the same line as code -> ignore
    textLine = document.line(line);
    if (textLine.search(/^\s*;;;/) != -1) {
        return 0;
    } else if (textLine.search(/^\s*;;/) != -1) {
        // try to align with the next line
        nextLine = document.nextNonEmptyLine(line + 1);
        if (nextLine != -1) {
            return document.firstVirtualColumn(nextLine);
        }
    }

    cursor = document.anchor(line, 0, '(');
    if (cursor) {
        return document.toVirtualColumn(cursor.line, cursor.column) + indentWidth;
    } else {
        return 0;
    }
}

The result:

;;; fib : number -> number
(define (fib n)
  (if (< n 2)
    1
    (+ (fib (- n 1)) (fib (- n 2)))))

As this indenter is scripted, everybody can adapt the style to the own needs.

Kate Scripting: Indentation

Kate Part in KDE4 supports the ECMAScript (JavaScript) language by using kjs. In KDE3 we had several hard-coded indenters in C++, the idea is to let scripts do all the indentation in KDE4.
How does it work? It is similar to vim: You simply create a script in the directory $KDEDIR/share/apps/katepart/jscript. An indentation script has to follow several rules:

1. it must have a valid script header (the first line must include the string kate-script and indentation scripts must have the type: indentation)
   
2. it must define some variables and functions

Whenever the user types a character, the flow in Kate Part works like this

1. check the indentation script's trigger characters, i.e. whether the script wants to indent code for the typed character
2. if yes, call the indentation function
3. the return value of the indentation function is an integer value representing the new indentation depth in spaces.

In the 3rd step there are 2 special cases for the return value:

1. return value = -1: Kate keeps the indentation, that is it searches for the last non-empty line and uses its indentation for the current line
2. return value < -1 tells Kate to do nothing

So how does a script look like exactly?
The name does not really matter, so let's call it foobar.js:

/* kate-script
* name: MyLanguage Indenter
* license: LGPL
* author: Foo Bar
* version: 1
* kate-version: 3.0
* type: indentation
*
* optional bla bla here
*/

// specifies the characters which should trigger indent() beside the default '\n'
triggerCharacters = "{}";

// called for the triggerCharacters {} and
function indent(line, indentWidth, typedChar)
{
// do calculations here
// if typedChar is an empty string, the user hit enter/return

// todo: Implement your indentation algorithms here.
return -1; // keep indentation
}

More details on the header:

• name [required]: the name will appear in Kate's menus
• license [optional]: not visible in gui, but should be specified in js-files. it is always better to have a defined license
• author [optional]: name
• version [optional]: recommended. an integer
• kate-version [required]: the minimum required kate-version (e.g. for api changes)
• type [required]: must be set to indentation

The only missing part is the API which Kate Part exports to access the document and the view. Right now, there is no API documentation, so you have to look at the code:

• kdelibs/kate/jscript/katejscript.cpp search for "@Begin" tags
  

You will find, that the current document exports functions like

• document.fileName()
• document.isModified()
• document.charAt(line, column)
• etc...

The view exports functions like

• view.cursorPosition()
• view.hasSelection()
• view.clearSelection()
• ...

That's the boring part of this blog. The interesting one is unfortunately shorter: we are looking for contributors who want to write scripts or help in the C++ implementation :)