Friday, July 09, 2010
Moving my Blog
Kate: Scripted Actions
What happens is the following:/* kate-script
* author: Dominik Haumann
* license: LGPL
* revision: 3
* kate-version: 3.4
* type: commands
* functions: moveLinesDown
*/
function moveLinesDown()
{
var fromLine = -1;
var toLine = -1;
var selectionRange = view.selection();
if (selectionRange.isValid() && selectionRange.end.line < document.lines() - 1) {
toLine = selectionRange.start.line;
fromLine = selectionRange.end.line + 1;
} else if (view.cursorPosition().line < document.lines() - 1) {
toLine = view.cursorPosition().line;
fromLine = toLine + 1;
}
if (fromLine != -1 && toLine != -1) {
var text = document.line(fromLine);
document.editBegin();
document.removeLine(fromLine);
document.insertLine(toLine, text);
document.editEnd();
}
}
function action(cmd)
{
var a = new Array();
if (cmd == "moveLinesDown") {
a['text'] = i18n("Move Lines Down");
a['icon'] = "";
a['category'] = "";
a['interactive'] = false;
a['shortcut'] = "";
}
return a;
}
function help(cmd)
{
if (cmd == "moveLinesDown") {
return i18n("Move selected lines down.");
}
}
- the header tells kate that there is an exported function "moveLinesDown"
- so when Kate Part is loaded, it calls "action(moveLinesDown)" to check whether this function should be exposed in the GUI. Here, we return the action info that includes the displayed text, an icon, a category, whether the script needs user input (interactive) and a default shortcut. Of course, you can change the shortcuts, and also configure the toolbars to show the actions.
You can have this already now, you just have to use the development version of Kate :)
Wednesday, July 07, 2010
Enjoying Tampere
Thursday, June 24, 2010
A Flashback of Kate in Gitorious
As getting new contributors is essential for keeping a project alive, the barrier to get involved should be as low as possible. And exactly this was achieved by moving all pieces to one place (this was gitorious for us). Building Kate is so simple right now that we can even make bug reporters build Kate out of the box. This helps a lot, and even results in patches from time to time. We also got quite some merge requests.
There were several voices at that time that considered moving "away from KDE" was very bad. However, this is not the case, as Christoph is synchronizing all the changes in KDE's subversion and gitorious almost every day. This is certainly not optimal, but looking back at the last months, we can say it was worth it.
KDE is moving to git.kde.org in the near future. This also raises the discussion about how KDE's source code will be organized. Speaking for Kate, we certainly want to have all of Kate's code in one place, just as it is now with gitorious, no matter what :) I hope we can find a solution the KDE community can live with. To be discussed, maybe in Tampere in two weeks? :)
Saturday, June 19, 2010
Kate: Debugging with Qt-Creator
- build Kate according to this tutorial
- install Qt-Creator (in my case this is version 2.0.0rc1)
- start Qt-Creator: ~/kde/run.sh /path/to/qtcreator/bin/qtcreator
- invoke File > Open File or Project and choose ~/kde/kate/CMakeLists.txt
- Build Location: choose ~/kde/build
- Run CMake arguments: choose ../kate -DCMAKE_BUILD_TYPE=debugfull -DCMAKE_INSTALL_PREFIX=~/kde/usr
- click the button "Run CMake" and then "Finish"
- click the "hammer" icon button on the very bottom left to compile Kate
- click the "computer" icon button (3 icons above the hammer icon) and choose "kate" (or "kwrite") in the Run combo box
- choose the "Debug" icon in the left pane
- invoke "Debug > Start Debugging (F5)", now Kate starts
- open part/document/katedocument.cpp in the file tree view on the left
- go to the line "KateDocument::insertText" and click "Debug > Toggle Breakpoint (F9)"
- now if you type a character in Kate, Qt-Crator will halt in KateDocument::insertText
- chose "Debug > Step Opver (F10)" (and "Debug > Stip Into (F11)") to step through the code
- click on "Locals and Watchers" in the debugging pane on the bottom and you see the values of local variables
Thursday, June 10, 2010
Kate: Code Folding Crash
Update: Fixed by Stefan Schenk in this commit for KDE 4.5. Awesome! :)
Wednesday, April 28, 2010
Kate Internals: Smart Cursors and Smart Ranges
Since KDE 4.0 the KTextEditor interfaces have so called SmartCursors and SmartRanges. A SmartCursor is a text cursor (i.e. a line/column tuple) , which is bound to a text document. When editing the text document, the cursor is automatically moved such that it maintains its position. You need this for the displayed text cursor in a view for instance. If you type text, the cursor automatically moves.
A SmartRange consists of two SmartCursors: start and end. We use that for instance for the text selection or the inline spell checking, or KDevelop uses it to add arbitrary highlighting to parsed C/C++ code. Again, if you modify text in the document, the text range automatically moves, expands or shrinks.
The concept of SmartCursors and SmartRanges is doubtless very useful. However, for KDE 4.5 the Kate developers came up with an alternative implementation that will even deprecate the SmartCursors and SmartRanges in KDE 4.6. The reason for this is that the current implementation has several problems, which we will discuss in the following.
API Usage
The SmartRanges API can be used in very wrong ways. For instance, the SmartInterface has the two functions deleteCursors() and deleteRanges(). Besides that, a document reload also deletes all SmartCursors and SmartRanges. Both cases lead to a dangling pointer if you use SmartCursors or SmartRanges. Hence, whenever you use them, you always need a so-called SmartCursorWatcher / SmartCursorNotifier and SmartRangeWatcher / SmartRangeNotifier, which tell you that your SmartCursor or SmartRange is about to be deleted. Watcher and Notifier basically do the same thing. Besides deletion notification, they also tell you more e.g. that the SmartCursor position changed. If you use a notifier, those notifications are sent via signals. If you use a watcher, you have to overwrite virtuals. That is, we have two different concepts that basically do the same thing. If you use thousands of SmartRanges e.g. for arbitrary highlighting, we have possibly thousands of those emits, which does not really scale.
The API also allows to have parent/child relations, i.e. a SmartRange can have children and a parent. It is possible to traverse this hierarchy, and again since you get the pointers you can delete arbitrary SmartRanges or SmartCursors. Another reason why you always need a notifier or watcher. And if you have a notifier/watcher, you always have the signals emitted when e.g. a cursor changes, as explained above.
Further, we have lots of places where we use reference object KTextEditor::Cursor& as parameter. This const reference-object signals that it does not change. But due to SmartRange deriving from KTextEditor::Cursor, this object might also be a SmartCursor. So despite of being declared as a const object, it may change its position behind your back. This may lead to unexpected behaviours in e.g. loops where the line or column of the cursor is assumed to be constant. This is a problem in almost all functions in KatePart, so passing SmartCursors as Cursors is a very bad idea, but of course allowed.
Scaleability Problems
As mentioned above, you always need a Notifier or Watcher for all SmartCursors and SmartRanges. This will emit lots of signals you probably rarely need. Still, there is no way around it. This is a huge overhead, it simply does not scale.
Implementation
The implementation is really complex and rather undocumented in most areas. Unfortunately, only very few (or maybe even no one) really understand the code. Besides that, it is very fragile. If you change something, you might break the code. Hence, we often heard similar comments like "don't touch the code, otherwise it will break". Such comments alone already indicate that the current implementation is not maintainable. Unmaintainable code is bad, especially in open source projects (this is not a good way to gain new developers at all). There are other downsides of the current implementation: SmartCursors need a lot of memory for each instance; there are lots of features like animations for SmartRanges, which make the code even more complex; redundancy of watchers and notifiers bloat the code and influence the runtime behavior.
Threading
It seems there was the idea of making the KTextEditor interfaces thread safe. The SmartRanges interact with the content of the document, e.g. querying lines count and line length of lines in the text buffer. As this is done by other threads we need correct locking in all places: in document content changes, in smart cursors, in smart ranges, in the painting routines, etc. The current state in KatePart is that not all functions do the right locking. That's why we have lots of asserts/crashs. KatePart already has more than 150 lockings at the moment, but they still do not voer problems. And very few developers (or no one?) really know when to lock and when not to. This is especially complex since we want to ensure that the locks are not hold while emitting signals or calling functions provided from the outside as callbacks, this is still not done completely right, too.
If you think about Qt, the GUI widgets are also not thread-safe, they all live in the main thread. And if you need data in other threads, you always use queued signals. This is pretty much what we experience in KatePart now. Not enough locking? Too much locking? In other words: It's far too complex to make the KTextEditor interfaces thread-safe...
Threading and Revisions
Now to another issue in the implementation. KDevelop uses KTextEditor::Ranges in the parsing thread. Think of the following use case: KDevelop gets all the document text in the current revision (or version, if you prefer). Now it takes some time until the parsing is finished. When it's finished, KDevelop uses the Ranges (which initially belong to the old revision of the document), and then translates those ranges to the current version of the text document (this is simple: Just track all edits inside KatePart (delete, insert, wrap, unwrap) and apply those transformations to the ranges). Now we transformed ranges from an old text revision to the current text revision. This means KDevelop does not have to parse everything again, as it knows exactly which parts changed. Awesome indeed :) However, now comes the problem: To transform Cursors and Ranges between revisions, you have to tell the SmartInterface about the revision you are working with. This is done via SmartInterface::useRevision(int). The API documentation says: Tell the smart interface to work against the given revision when creating cursors and ranges. That is, if you call useRevision() once, all succeeding calls like newSmartRange() etc are working in the past (at that revision). Also kind of cool, since useRevision() works locally in each thread. That is different threads don't influence each other. But there is huge problem with this: Think of a KTextEditor user (KDevelop, Kile, ...) uses useRevision() in the main thread. Then all succeding calls of e.g. newSmartRange() use an old revision instead of the current one. Hence, KatePart itself is completely broken in that case. (This case is now catched via a qFatal() in KatePart). But again this shows that multi-threading simply complicates the matter a lot. It would have been much easier to say transformRange(int fromRevision, int toRevision) instead of just one translateFromRevision() that translates the given range against the revision specified through useRevision(). Hence, the current API is unfortunately pretty much broken by design.
A necessary Step?
Often it's hard to get things done right in the first try. So maybe the issues above are a necessary step to a much better implementation. And that is exactly what happened in the last two months: We have new interfaces called MovingInterface, MovingCursor and MovingRange. For feedback there is MovingRangeFeedback, which uses callbacks for notification (no signals by intention). All those classes have unit tests. KatePart in KDE 4.6 will not implement any Smart* interface anymore. Hence, KatePart will be completely single threaded again. We already started to remove several thousands of lines Smart* code that was never used/finished. Much more will follow immediately after the KDE 4.5 release. This means all developers using Smart* from the KTextEditor interfaces should migrate to the Moving* pendents for KDE 4.5 already, if possible.
I'll provide very detailed information about the Moving* implementation in further blogs during the next month.
Tuesday, April 27, 2010
Kate XML Completion Plugin
Saturday, April 17, 2010
Quick Compiling Kate in a stable KDE Environment
- make sure you have the following packages installed: git, cmake and kdelibs development package (on openSUSE this is git, cmake and libkde4-devel)
- create and change into a KDE development directory:
mkdir ~/kde; cd ~/kde - get a copy of the Kate code:
git clone git://gitorious.org/kate/kate.git - create and change into a build directory for compilation:
mkdir build; cd build - run the configure process with cmake:
cmake ../kate -DCMAKE_BUILD_TYPE=fulldebug \
-DCMAKE_INSTALL_PREFIX=~/kde/usr - compile Kate:
make - finally install Kate:
make install
Now on to starting the newly compiled Kate. Create a file ~/kde/run.sh with the following content:
Now you can run the compiled Kate version with ~/kde/run.sh kate. Just calling kate directly will start the system version of Kate.#!/bin/bash
export KATE_DIR=~/kde/usr
export PATH=$KATE_DIR/bin:$PATH
export LD_LIBRARY_PATH=$KATE_DIR/lib:$LD_LIBRARY_PATH
export KDEDIR=$KATE_DIR
export KDEDIRS=$KDEDIR
export XDG_DATA_DIRS=$XDG_DATA_DIRS:$KATE_DIR/share
# update KDE's system configuration cache
kbuildsycoca4
# start app
$@
Your copy of Kate contains all of the Kate code, i.e.: the KTextEditor interfaces, Kate Part, KWrite and the Kate Application. Feel free to send patches to our mailing list kwrite-devel@kde.org. And join #kate on irc.kde.org :-)
Note for KDE developers: All the changes in git are merged back to KDE's subversion repository in a timely manner. So don't worry about Kate moving away from KDE; this is not the case.
(Updated on 2010-04-17: Allow installation into local directory.)
Tuesday, March 02, 2010
Kate Internals: Text Buffer
Right now, in Kate's gitorious repository we, or rather mainly Christoph, is rewriting the heart of Kate: The text buffer. In this blog I'll explain how the text buffer works in detail. After reading, you'll be able to understand the code. And be warned: Probably you will be desperately amazed about its awesomeness! So this is a must read :^) The following content is based on the notes for design ideas for the new text buffer.
Storage. Kate Part internally stores the content of a text document as a QVector of text lines. Each text line holds the text in a QString object as well as additional data like color attributes for highlighting. Inserting and removing text lines implies inserting or removing items in the QVector of the text lines. This is can get slow when there are thousands of text lines in a document, because QVector has to move the memory every time. Thus, to keep text operations fast, Kate splits all text lines into several text blocks: Each text block contains a certain amount of lines (e.g. 256). The expense of memory movement is then always limited. When a text block grows too much, it is automatically split. When the amount of lines in a text block shrinks, it is merged with the previous text block. In other words, Kate's text buffer automatically balances the text blocks. The basic idea of Kate's text buffer, the text blocks and text lines looks like this:
Text Cursors and Ranges. Text manipulation always takes place at certain positions in a text. Such a position is called a text cursor. Each text cursor is defined by a line and a column. Further, to specify e.g. text selections, we need text ranges. A text range has a start cursor and an end cursor. For instance, if you have two views of the same document, you want the cursor in the 2nd view to keep its correct position in the text if you insert text in the 1st view. Thus, text cursors have to be kind of intelligent. Whenever text changes, each text cursor needs to be moved. As a text range just consists of two text cursors, we will focus only on text cursors for now. The question is how to implement this efficiently? If we store all text cursors in a list in the text buffer, we have to iterate over all text cursors on every editing operation. This obviously does not scale for thousands of text cursors (KDevelop for instance uses thousands of text cursors for the highlighting). The solution is the same as with the text content itself: Let's just put a list of all text cursors in a text block to the text block itself. During text editing, we only have to adapt all text cursors of a single text block instead of all text cursors in a document. This looks as follows:
Editing Operations. When editing text, you usually have only four different types of text manipulation:
- insert text at a text cursor inside one line
- remove text at a text cursor inside one line
- wrap line at a text cursor (e.g. by hitting <return/enter>)
- unwrap line (e.g. by hitting <backspace> at the beginning of a line)
Transactions. A transaction consists of several of the four editing operations. For instance, if you select text and then move it to another location in the document with drag & drop you want this to be grouped together to a single text operation on undo/redo. (The text operations in this case are unwrap lines and remove text, then insert text and wrap line). To be able to specify which parts of text operations belong together, the text buffer provides two functions: startEditing() starts a transaction and finishEditing() closes it. Those functions use reference counting, so you can call startEditing() multiple times and only the last finishEditing() completes a transaction. Again, signals are emitted e.g. in finishEditing() such that other layers (undo/redo system) are notified about this.
Revisions. As an easy way to check whether a document changed you can get the current revision of the text buffer. The revision is simply an int64 starting with 0 after loading a document. The revision is incremented in every of the 4 editing primitives. This way you don't have to listen to multiple signals like textInserted() and textRemoved(). This could be useful for e.g. KDevelop's background parser: Parse a file. When done, check whether the revision is the same. If yes, parsing was successful. If not, parse again. This is how QtCreator does it. Easy and straight forward.
Unit Tests. The text buffer, text block, text line, text cursors are all designed such that unit tests can be written. Hence, each of the above implementation details will be covered by unit tests.
Further ideas. As mentioned previously, the design of the text buffer leaves room for further features. For now, let's face two ideas:
- Vim like swap files: If Kate or the application using Kate Part crashes, the unsaved data in the text document is lost. Vim has a backup mechanism called 'swap files' to recover from crashes to avoid data loss since years, and it works as follows: Each of the 4 editing primitives explained above are streamed into a separate file. When you save a file successfully, the swap file is cleared. If a crash happens, vim loads the original text document and then replays all editing operations in the swap file. This way the current unsaved editing state is recovered; no data is lost. Since the text buffer emits all necessary signals, implementing this feature is kind of trivial. Any takers?
- Right now, the highlighting works in an extra class, accessing the text buffer's text lines and setting the text attributes such as the color. As idea, we could also derive a class Kate::HighlightingBuffer from Kate::TextBuffer and do the additional modifications in the derived class instead of an extra class. We'll have to see whether that makes sense.
Please Contribute.
- join #kate on irc://irc.kde.org
- checkout the code base, clone it, create branches, and merge back!
- other kate internals: undo/redo system (little bit outdated, but all in all still valid)
Saturday, February 27, 2010
Kate Partly Moving to Gitorious
We have discussed a bit about the migration to gitorious on the Kate Developer Meeting and Christoph came up with this mainly because building only KTextEditor, Kate Part, KWrite and Kate is much faster and easier compared to building the KDE modules kdesupport, kdelibs, kdepimlibs, kdebase, kdesdk.
I myself remember the time where I started KDE development, and it took more than two weeks to have a first successful build of KDE. You have to learn so many things at once, like revision control, lots of so far unknown software, and what not. Talking to other developers verifies this. In other words: Getting into KDE development is not easy and straight forward.
Moving to gitorious removes this barrier for Kate development: You just checkout the Kate repository and that's all you need. It would be nice if you join Kate development and contribute patches :)
What does that mean for Kate in KDE? Nothing changes. We will merge the changes in Gitorious back to the main KDE development line and vice versa.
Wednesday, February 17, 2010
The Power of Developer Meetings
- Social aspect: You get to know the other developers involved in the project in real life, which is a great motivation factor. This also happens at KDE's annual conference Akademy, although there are a lot more people.
- Productivity: Since you are sitting next to each other discussions about how to do what are very focused. It's amazing how quickly a project can evolve this way. (I haven't seen such focused work in companies, yet. I guess the variance in the knowledge of people involved is higher. And the motivation is usually very different).
- Knowledge Transfer: Since participants are experts in different areas, discussions lead to knowledge transfer. This is essential, as sometimes developers have very few free time to contributes to a project. Spreading the knowledge helps a lot to keep the project alive.
- Steady Contributions: We are always open for new contributors. Just send a patch, get commit access and join development. Experience shows that participants of a developer meeting usually contribute for years to come.
- memory optimization: remove unused edit history items
- fix crash in doHighlight due to wrong initialization
- don't squeeze captions in the window title
- fix crash when using katepart in e.g. konqueror
- kate vi mode improvements
- fix: invalid js commands lead to a crash
- share code snippets between kate and kdevelop
- more syntax highlighting files for asm dialects
- replace: show number of replacements in a passive popup
- new command line command: unwrap
- fix crash when switching sessions with multiple main windows
- fix incorrect global replace %s///g
- fix crash in smart range group handling
- turn search option "highlight all" into the action "find all"
- replace old text snippet plugin with new one
- optimize away nops for text editing
- fixes in the template handler code
- remove pythong browser plugin, since the symbol viewer plugin can do the same and much more
- remove unmaintained html tools plugin, since it was never ported
- optimization for bracket matching
- new text snippet plugin: context aware snippet completion
- fix filtering in the file browser plugin
- always start search&replace from cursor
- developer doumentation for smart range usage
- cstyle indenter: fix indentation of multi-line comments
- update of syntax highlighting for MySQL
- performance optimization in the highlighting system: bug, bug
- fix crash in removeTrailingSpaces, if undo/redo is active
- new interface for getting the mode at a given cursor
- search & replace: lots of code cleanups
- basic regression tests for search, scripting, and more...
- scripting: add support for quoting in command line scripts
- Kate crashes because of spell checking
- fix regression: with auto brackets + selection
- find-in-files: set filter correctly when opening the search
- fix: make reload-scripts work with new functions
- share a KCompletion object between all KateCmdLineEdits
Developer Meeting: More on scripting Kate
Yesterday I closed a bug requesting an "unwrap" feature in Kate that works like "Tools > Join Lines" but maintains paragraph separation, i.e., empty lines are not removed. This feature is implemented now in javascript. Further infos:
To run the script simply switch to the command line (F7) and write "unwrap". If you have further ideas about useful scripts, don't hesitate to start hacking right away, see also
Fixes with regard to the scripting support in the last days are
Those fixes will be in KDE 4.4.1. More to come in other blog entries :-)
Sunday, January 17, 2010
Kate XML Completion Plugin: Help Wanted
Unfortunately, this plugin has not been ported to KDE4, yet. So this is a request for developers: Please someone pick it up and make it available for Kate in KDE4.
The location in svn is: trunk/KDE/kdesdk/kate/plugins/xmltools
Just enable this folder in the kate/plugins/CMakeLists.txt file and start porting it. The code completion interface changed quite a lot, but there are other plugins and good api documentation (KTextEditor interfaces, Kate Application interfaces) where you can have a look at how things work.
Having a working XML Completion plugin for KDE4.5 would be awesome.