Moving my Blog

Since we have a new Kate Homepage, I''ve moved all my content over. Now the kate homepage at least has lots of Kate related content :)

Kate: Scripted Actions

Finally, I came around to implement scripted actions for Kate in KDE SC 4.6. Let's take a look at how this works. When Kate starts, it searches for $KDEDIRS/share/apps/katepart/script/ for *.js files. As example, let's take a look at utils.js there:

/* kate-script
* author: Dominik Haumann
* license: LGPL
* revision: 3
* kate-version: 3.4
* type: commands
* functions: moveLinesDown
*/

function moveLinesDown()
{
  var fromLine = -1;
  var toLine = -1;

  var selectionRange = view.selection();
  if (selectionRange.isValid() && selectionRange.end.line < document.lines() - 1) {
      toLine = selectionRange.start.line;
      fromLine = selectionRange.end.line + 1;
  } else if (view.cursorPosition().line < document.lines() - 1) {
      toLine = view.cursorPosition().line;
      fromLine = toLine + 1;
  }

  if (fromLine != -1 && toLine != -1) {
      var text = document.line(fromLine);

      document.editBegin();
      document.removeLine(fromLine);
      document.insertLine(toLine, text);
      document.editEnd();
  }
}

function action(cmd)
{
  var a = new Array();
  if (cmd == "moveLinesDown") {
      a['text'] = i18n("Move Lines Down");
      a['icon'] = "";
      a['category'] = "";
      a['interactive'] = false;
      a['shortcut'] = "";
  }

  return a;
}

function help(cmd)
{
  if (cmd == "moveLinesDown") {
      return i18n("Move selected lines down.");
  }
}

What happens is the following:

1. the header tells kate that there is an exported function "moveLinesDown"
2. so when Kate Part is loaded, it calls "action(moveLinesDown)" to check whether this function should be exposed in the GUI. Here, we return the action info that includes the displayed text, an icon, a category, whether the script needs user input (interactive) and a default shortcut. Of course, you can change the shortcuts, and also configure the toolbars to show the actions.
   

With this, every user is able to script arbitrary editing functions for Kate Part. We don't have to implement all those helpers in C++ anymore. The result looks like this:
You can have this already now, you just have to use the development version of Kate :)

Enjoying Tampere

As always, the KDE conference has its funny sides, as you can see on the photo: 4 Kate developers shaping the future of the (as you all know) most awesome application: Kate :-)

A Flashback of Kate in Gitorious

Back in February, I blogged about Kate's move to gitorious. The main reason for this move was to make building Kate as easy as possible. If you want to build Kate as part of KDE, (as of now) you have to compile kdesupport, phonon, dbusmenu-qt, kdelibs, kdepimlibs, kdebase for kwrite and kdesdk for the kate application. Getting all this done is a huge effort, especially if you are new to KDE development (I very well remember my own times spending weeks to get everything going. Be aware of new contributors might now close to nothing about KDE and all the dependencies!).
As getting new contributors is essential for keeping a project alive, the barrier to get involved should be as low as possible. And exactly this was achieved by moving all pieces to one place (this was gitorious for us). Building Kate is so simple right now that we can even make bug reporters build Kate out of the box. This helps a lot, and even results in patches from time to time. We also got quite some merge requests.
There were several voices at that time that considered moving "away from KDE" was very bad. However, this is not the case, as Christoph is synchronizing all the changes in KDE's subversion and gitorious almost every day. This is certainly not optimal, but looking back at the last months, we can say it was worth it.
KDE is moving to git.kde.org in the near future. This also raises the discussion about how KDE's source code will be organized. Speaking for Kate, we certainly want to have all of Kate's code in one place, just as it is now with gitorious, no matter what :) I hope we can find a solution the KDE community can live with. To be discussed, maybe in Tampere in two weeks? :)

Kate: Debugging with Qt-Creator

Let's have a quick look at how to debug Kate and KWrite with Qt-Creator. First, make sure you meet the requirements:

1. build Kate according to this tutorial
2. install Qt-Creator (in my case this is version 2.0.0rc1)

Setup the Kate project Qt-Creator once like this

1. start Qt-Creator: ~/kde/run.sh /path/to/qtcreator/bin/qtcreator
   
2. invoke File > Open File or Project and choose ~/kde/kate/CMakeLists.txt
3. Build Location: choose ~/kde/build
4. Run CMake arguments: choose ../kate -DCMAKE_BUILD_TYPE=debugfull -DCMAKE_INSTALL_PREFIX=~/kde/usr
5. click the button "Run CMake" and then "Finish"

Start debugging like this

1. click the "hammer" icon button on the very bottom left to compile Kate
2. click the "computer" icon button (3 icons above the hammer icon) and choose "kate" (or "kwrite") in the Run combo box
3. choose the "Debug" icon in the left pane
4. invoke "Debug > Start Debugging (F5)", now Kate starts
   
5. open part/document/katedocument.cpp in the file tree view on the left
6. go to the line "KateDocument::insertText" and click "Debug > Toggle Breakpoint (F9)"
7. now if you type a character in Kate, Qt-Crator will halt in KateDocument::insertText
8. chose "Debug > Step Opver (F10)" (and "Debug > Stip Into (F11)") to step through the code
9. click on "Locals and Watchers" in the debugging pane on the bottom and you see the values of local variables

Happy debugging :)

Kate: Code Folding Crash

We still have a crash in Kate's code folding code; no one was able to find a proper fix, yet. So if you want to get your hands dirty, just build Kate, find a fix and be the hero of all Kate developers :)

Update: Fixed by Stefan Schenk in this commit for KDE 4.5. Awesome! :)

Kate Internals: Smart Cursors and Smart Ranges

SmartCursors and SmartRanges in KDE 4.0 - KDE 4.4

Since KDE 4.0 the KTextEditor interfaces have so called SmartCursors and SmartRanges. A SmartCursor is a text cursor (i.e. a line/column tuple) , which is bound to a text document. When editing the text document, the cursor is automatically moved such that it maintains its position. You need this for the displayed text cursor in a view for instance. If you type text, the cursor automatically moves.
A SmartRange consists of two SmartCursors: start and end. We use that for instance for the text selection or the inline spell checking, or KDevelop uses it to add arbitrary highlighting to parsed C/C++ code. Again, if you modify text in the document, the text range automatically moves, expands or shrinks.

The concept of SmartCursors and SmartRanges is doubtless very useful. However, for KDE 4.5 the Kate developers came up with an alternative implementation that will even deprecate the SmartCursors and SmartRanges in KDE 4.6. The reason for this is that the current implementation has several problems, which we will discuss in the following.

API Usage

The SmartRanges API can be used in very wrong ways. For instance, the SmartInterface has the two functions deleteCursors() and deleteRanges(). Besides that, a document reload also deletes all SmartCursors and SmartRanges. Both cases lead to a dangling pointer if you use SmartCursors or SmartRanges. Hence, whenever you use them, you always need a so-called SmartCursorWatcher / SmartCursorNotifier and SmartRangeWatcher / SmartRangeNotifier, which tell you that your SmartCursor or SmartRange is about to be deleted. Watcher and Notifier basically do the same thing. Besides deletion notification, they also tell you more e.g. that the SmartCursor position changed. If you use a notifier, those notifications are sent via signals. If you use a watcher, you have to overwrite virtuals. That is, we have two different concepts that basically do the same thing. If you use thousands of SmartRanges e.g. for arbitrary highlighting, we have possibly thousands of those emits, which does not really scale.

The API also allows to have parent/child relations, i.e. a SmartRange can have children and a parent. It is possible to traverse this hierarchy, and again since you get the pointers you can delete arbitrary SmartRanges or SmartCursors. Another reason why you always need a notifier or watcher. And if you have a notifier/watcher, you always have the signals emitted when e.g. a cursor changes, as explained above.

Further, we have lots of places where we use reference object KTextEditor::Cursor& as parameter. This const reference-object signals that it does not change. But due to SmartRange deriving from KTextEditor::Cursor, this object might also be a SmartCursor. So despite of being declared as a const object, it may change its position behind your back. This may lead to unexpected behaviours in e.g. loops where the line or column of the cursor is assumed to be constant. This is a problem in almost all functions in KatePart, so passing SmartCursors as Cursors is a very bad idea, but of course allowed.

Scaleability Problems

As mentioned above, you always need a Notifier or Watcher for all SmartCursors and SmartRanges. This will emit lots of signals you probably rarely need. Still, there is no way around it. This is a huge overhead, it simply does not scale.

Implementation

The implementation is really complex and rather undocumented in most areas. Unfortunately, only very few (or maybe even no one) really understand the code. Besides that, it is very fragile. If you change something, you might break the code. Hence, we often heard similar comments like "don't touch the code, otherwise it will break". Such comments alone already indicate that the current implementation is not maintainable. Unmaintainable code is bad, especially in open source projects (this is not a good way to gain new developers at all). There are other downsides of the current implementation: SmartCursors need a lot of memory for each instance; there are lots of features like animations for SmartRanges, which make the code even more complex; redundancy of watchers and notifiers bloat the code and influence the runtime behavior.

Threading

It seems there was the idea of making the KTextEditor interfaces thread safe. The SmartRanges interact with the content of the document, e.g. querying lines count and line length of lines in the text buffer. As this is done by other threads we need correct locking in all places: in document content changes, in smart cursors, in smart ranges, in the painting routines, etc. The current state in KatePart is that not all functions do the right locking. That's why we have lots of asserts/crashs. KatePart already has more than 150 lockings at the moment, but they still do not voer problems. And very few developers (or no one?) really know when to lock and when not to. This is especially complex since we want to ensure that the locks are not hold while emitting signals or calling functions provided from the outside as callbacks, this is still not done completely right, too.

If you think about Qt, the GUI widgets are also not thread-safe, they all live in the main thread. And if you need data in other threads, you always use queued signals. This is pretty much what we experience in KatePart now. Not enough locking? Too much locking? In other words: It's far too complex to make the KTextEditor interfaces thread-safe...

Threading and Revisions

Now to another issue in the implementation. KDevelop uses KTextEditor::Ranges in the parsing thread. Think of the following use case: KDevelop gets all the document text in the current revision (or version, if you prefer). Now it takes some time until the parsing is finished. When it's finished, KDevelop uses the Ranges (which initially belong to the old revision of the document), and then translates those ranges to the current version of the text document (this is simple: Just track all edits inside KatePart (delete, insert, wrap, unwrap) and apply those transformations to the ranges). Now we transformed ranges from an old text revision to the current text revision. This means KDevelop does not have to parse everything again, as it knows exactly which parts changed. Awesome indeed :) However, now comes the problem: To transform Cursors and Ranges between revisions, you have to tell the SmartInterface about the revision you are working with. This is done via SmartInterface::useRevision(int). The API documentation says: Tell the smart interface to work against the given revision when creating cursors and ranges. That is, if you call useRevision() once, all succeeding calls like newSmartRange() etc are working in the past (at that revision). Also kind of cool, since useRevision() works locally in each thread. That is different threads don't influence each other. But there is huge problem with this: Think of a KTextEditor user (KDevelop, Kile, ...) uses useRevision() in the main thread. Then all succeding calls of e.g. newSmartRange() use an old revision instead of the current one. Hence, KatePart itself is completely broken in that case. (This case is now catched via a qFatal() in KatePart). But again this shows that multi-threading simply complicates the matter a lot. It would have been much easier to say transformRange(int fromRevision, int toRevision) instead of just one translateFromRevision() that translates the given range against the revision specified through useRevision(). Hence, the current API is unfortunately pretty much broken by design.

A necessary Step?

Often it's hard to get things done right in the first try. So maybe the issues above are a necessary step to a much better implementation. And that is exactly what happened in the last two months: We have new interfaces called MovingInterface, MovingCursor and MovingRange. For feedback there is MovingRangeFeedback, which uses callbacks for notification (no signals by intention). All those classes have unit tests. KatePart in KDE 4.6 will not implement any Smart* interface anymore. Hence, KatePart will be completely single threaded again. We already started to remove several thousands of lines Smart* code that was never used/finished. Much more will follow immediately after the KDE 4.5 release. This means all developers using Smart* from the KTextEditor interfaces should migrate to the Moving* pendents for KDE 4.5 already, if possible.

I'll provide very detailed information about the Moving* implementation in further blogs during the next month.

Kate XML Completion Plugin

A while ago I've blogged about the Kate's most powerful plugins: the XML completion plugin. Thanks to Tomáš Trnka this plugin is back alive with all its features since today and compiled by default in trunk: It will be included in KDE 4.5. Yay! :) And if you need it already now for KDE 4.4, you can build Kate yourself very easily according to this howto, without touching your global KDE installation (probably interesting for KDE documentation writers!). Have fun!