Using Cygwin Bash as your Win32-Vim shell

Because the standard Vim that comes with Cygwin is kind of weak and nobody likes running an X server on his Windows computer, running the Win32 build of Vim in conjunction with Cygwin is a logical thing to do.

Although not ideal, it is possible to do this with an out-of-the-box Vim, by entering a few lines of code to your .vimrc. The problem with this is that complicated stuff, like for example this line out of my own .vimrc won't work; entering the :make command in Vim will give you errors:

autocmd BufRead *.py set makeprg=python\ -c\ 
       \"
          import\ py_compile,sys;\ sys.stderr=sys.stdout;\ 
          py_compile.compile(r'%')
       \"

(Without the newlines). Other issues with this method include that you can't use them in combination with PuTTYCyg, the best terminal for Cygwin you can find.

Now, I don't know about you, but "most commands" isn't nearly good enough for me. Ladies and Gentlemen, I give you: Vimshell.

Vimshell is a drop-in-place replacement for the vimrun.exe executable in your Vim directory (most of the time that would be C:\program files\Vim\vim72). You can find Vimshell right here.

Installation

First, you replace the standard vimrun.exe with the one you can find here. You then make a new file in the same folder that's called shell.txt. This file will contain exactly 3 lines. An example that works great with PuTTYCyg:

"C:\Program files\PuttyCyg\putty.exe" -cygterm "bash -i '#F;read;#'"
"C:\Program files\PuttyCyg\putty.exe" -cygterm "bash '#F#'"
"C:\Program files\PuttyCyg\putty.exe" -cygterm -

After Vimshell does its magic, the command that's going to be executed will look something like this:

"C:\Program files\PuttyCyg\putty.exe" 
            -cygterm "bash -i 'C:/somefile.txt'"

Where all commands your shell need to execute will be in the file C:/somefile.txt (so do whatever it takes to make sure your shell reads from whatever #F# will be replaced by). If you'd like to see the output of commands (and you probably do) you will need to make your shell pause after all commands have been executed. This is done by typing #F...something...# in your shell.txt instead of #F#. ...something... is the shell command that will be executed after everything else.

The first line in your shell.txt is the command that will be executed when a normal command needs to be executed by Vim (like when you type !echo "Hi"). The second line will be executed when a command is to be executed silently (like when you type :silent execute "!mkdir ~/temp 2>/dev/null" or :make). The third line will be executed when you just type :! (when you need an interactive shell).

Note that the functionality the second line provides is implemented in a much better way than it is by the standard vimrun.exe and the functionality the third line provides just isn't possible in a standard Vim+bash installation.

The installation is almost done, just add the following lines to your .vimrc file (this would be the .vimrc that's in your Cygwin home directory, not your Windows home directory; to check where this file is located enter the Vim command :echo $MYVIMRC):

set shellquote=
set shellslash  
set shellxquote=
set shellpipe=2>&1\|tee
set shellredir=>%s\ 2>&1

And that's it. Just use your Vim like you'd want to and execute command like :make at will.

How does it work?

At first, Vimshell worked by applying some complicated escaping rules that worked by scanning the strings in your shell.txt and then trying to escape your own commands in such a way that would give the correct result. This worked most of the time but unfortunately not always, since Bash does such strange things with arguments you give it and there's no way for me to predict how a string should be escaped. Also, Vimshell should work with any shell, not just bash, so this really wasn't a feasible option.

Thus, Vimshell currently employs another tactic. The command you wish to execute is written to a temporary file and this file is then given to and executed by your shell. That way, everything just works.

Putting the current directory in your terminal's titlebar: an optimization

Some time ago, I blogged about putting the current directory in your terminal's titlebar to improve your productivity. In the interest of dogfooding I used this myself for some time and I got annoyed by how much my terminal got slower.

The old (slow) code:

PROMPT_COMMAND="echo -ne \"\033]0;\$(pwd | sed "s#^$HOME#~#g")\007\""

Notice how the $(pwd | sed "s#^$HOME#~#g") part has to spawn at least two processes? One for pwd and another one for sed. Suffice it to say that spawning processes on Cygwin (Windows) isn't something you should take lightly.

Therefore, I propose to you the new and improved (fast) code:

PROMPT_COMMAND="echo -ne \"\033]0;\${PWD/\$HOME/~}\007\""

We replaced $(pwd | sed "s#^$HOME#~#g") with ${PWD/$HOME/~}. Instead of spawning 2 different process, this new command uses only bash built ins to accomplish the same thing. The bash syntax I'm using is quite obscure so let me quote the bash man pages (under Parameter Expansion):

${parameter/pattern/string}
The pattern is expanded to produce a pattern just as in pathname expansion. Parameter is expanded and the longest match of pat- tern against its value is replaced with string. If Ipattern begins with /, all matches of pattern are replaced with string. Normally only the first match is replaced. If pattern begins with #, it must match at the beginning of the expanded value of parameter. If pattern begins with %, it must match at the end of the expanded value of parameter. If string is null, matches of pattern are deleted and the / following pattern may be omit- ted. If parameter is @ or *, the substitution operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable sub- scripted with @ or *, the substitution operation is applied to each member of the array in turn, and the expansion is the resultant list.

Usage of this new syntax sped everything up a lot.

Interesting Problem: an image that's broken in Internet Explorer but works in every other browser

Earlier today, a friend contacted me with an interesting problem. He was working on a website that strangely enough had an image on it that was not visible in Internet Explorer but was visible in pretty much every other browser (Chrome, Firefox).

My first instinct was to blame this on a caching problem. Maybe he had visited the website in Chrome and Firefox earlier, and maybe they had cached it. He assured me this wasn't the case since clearing out all cached data didn't change a thing.

That was the point where I couldn't say anything more without looking at the page. I did and suddenly it became clear. The first part of the "problem" was that the page in question wasn't handwritten HTML, but was generated by Microsoft Excel. The HMTL code Excel generated for the image consisted of two distinct parts, some VML (Vector Markup Language) and a fallback to gif. Internet Explorer would try to read the VML file and pretty much every other browser would show the gif file. In itself this wouldn't be a problem since the emz file containing the compressed image data was present on the server. Which brings me to the second part of the problem.

This friend of mine wrote his own FTP program. Really. Even after I told him it was a bad idea (well, writing it isn't a bad idea but using it sure is). He uploaded the project folder using his own FTP program. His program tried to upload the emz file (filled with binary compressed data) as a text file, which obviously broke it (replacing \r\n with \n is not a good idea when processing binary files).

All of this results in Internet Explorer attempting to display the broken emz file (and failing) while every other browser ignores the emz file and correctly displays the gif file. I would not have guessed that without looking at the code...

Discreet & Private MSN Chatting from an Untrusted PC through a Unix Server with msnlib

Who doesn't like keeping in touch with his friends? Who doesn't like not being fired because your boss saw you launching Windows Live Messenger? Who doesn't like having a little privacy?

Your dreams aren't that far fetched! All you need is a UNIX (or Linux) server and a decent SSH client (PuTTY is a good candidate if you're on Windows). You don't need root privileges on either the server or your workstation.

The Server Setup

First, connect to your server. Download the latest version of msnlib on this website, for example with

wget http://blitiri.com.ar/p/msnlib/files/3.7/msnlib-3.7.tar.gz

Then extract the archive à la

tar xvzf msnlib-3.7.tar.gz
cd msnlib-3.7

There's no need to install, you just execute

bash msnsetup

and enter your Windows Live Passport email address and password when asked for.

Chatting Away

Connect to your server with SSH, cd your way to the directory in which you extracted msnlib and enter python msn.

You'll be greeted with a basic but functional TUI (Text User Interface). Type help to see what commands you can enter. You can press ee and hit ENTER to see who's online and m someone@something.com text to say "text" to someone@something.com. Reply to the last message your received with r text and send another message to the last person you sent one to with a text. To exit, just press CTRL+C.

FAQ

Why go through SSH? You can run <program> on your own computer!
If you run it on your own computer, all messages will be sent unencrypted over the company network.

Why aren't you using msncp/<other python based TUI msn client>?
There's nothing wrong with these, but they're mostly too graphical. Even though they run in a terminal, they still look like IM clients. People who see me using the msnlib msn client just back away slowly without blinking.

Advanced Vim: Recording & Editing Macros

Recording and Playing Macros

Macros are incredibly useful, as they allow you to automate certain annoying repetitive tasks. Let's jump right in.

You record macros by giving the qx command in normal (command) mode. Instead of x, you can pick any lowercase letter, so you can also give the command qb (record macro to register b) or qq (record macro to register q). This lower case letter identifies the register in which the macro will be recorded. If you record a macro in register b, you can play that macro by typing @b.

Enough of this boring talk, let's see an example I frequently use. Suppose you have some code that looks like this:

somefunction(paramA, paramB, paramC, paramD, paramE)

The parameters for this function have grown in size, maybe the line even exceeds 80 columns and you'd like each of the parameters to be on their own line. What I'd do is this (don't literally type [ENTER] or [ESC] but press the [ENTER] or [ESC] key).

qc                   Record the macro in register c
  f,                 Place the cursor right on the next comma
    a[ENTER]         Insert After the current character and press ENTER
            [ESC]q   Leave insert mode and stop recording	    

Place the cursor somewhere on the line you want to affect, before the first comma and press 4@c. This will execute the macro in register c four times. If you did everything correct this should give you something like

somefunction(paramA,
               paramB,
               paramC,
               paramD,
               paramE)

If either your Vim build doesn't support any auto indentation or there just isn't any for the current filetype, all of new lines will be misaligned and you will have to reindent manually (you can of course press TAB while recording macros, so repeatedly pressing tab after you hit [ENTER] while recording the macro would solve this problem but wouldn't work for lines of code that are indented either more or less). The macro will stay in register c until you either close your session or overwrite register c.

Editing recorded macros

Finally I can get to the advanced (and actually moderately interesting) part of this post. You probably know that one of the other places Vim uses registers is in it's clipboard management. If you want to yank (copy) a word into register a you give the "ayw (be sure to note the first character is actually a "! The double quote character at the start of a command tells Vim the next character will be a register.) command. To yank a line into register p, give the "pyy command. If you want to paste the contents of register a, you can do that by giving the "ap command.

Now here's the real kicker: the registers for copy paste and the registers for macros are the same registers! Think about that for a minute.

Back to our example, what if you suddenly realize you need to put a backslash at the end of every line (after every comma) so the compiler will know to look at the next line for the rest of this statement. You could re-record the entire macro, but if the macro was longer this wouldn't be too practical. The preferred course of action is to paste your macro right into the text you're writing, edit it and yank it back into its register.

We once again have the line

somefunction(paramA, paramB, paramC, paramD, paramE)

First, paste your macro by typing "cp (remember how we stored our macro in register c?). This will give you

f,a^M^[

The ^M is another way of writing the [ENTER] we pressed and ^[ is another way of writing [ESC]. We'd like to type a \ just before every newline (so right before ^M in our macro). So change the text we just pasted from register c to

f,a \^M^[

and put it back in register c by putting your cursor on the first character of the macro and typing "cy$ (yank everything to the end of the line into register c). Once again position yourself before the first comma and press 4@c

somefunction(paramA, \
                 paramB, \
                 paramC, \
                 paramD, \
                 paramE)

Perfect!

Putting the current directory in your terminal's titlebar on Linux & Cygwin

Having the current directory in your terminal's title bar has a number of advantages. If, like me, you have a lot (more than 5) of terminals open at any time, each in a different directory for a different project, you'll probably have to try several times before finding the correct window.

Most of this can easily be solved by adding a single line of code to your ~/.bashrc file. Let's see what that line of code is:

PROMPT_COMMAND="echo -ne \"\033]0;\$(pwd | sed "s#^$HOME#~#g")\007\""

There's a few things going on here so I'll about them one by one.

First of all, notice how I'm setting the PROMT_COMMAND bash variable. The contents of this variable are executed just after the command is executed and just before bash prints it's next prompt.

The command that's executed before every prompt, echo's a very special string. At the start of this string is an incantation that tells your terminal emulator to use the following text as the window title.

We'd like the current directory (given by the pwd command) to be shown, but if we're in a subdirectory of our home directory, we don't like to see the current path in it's rough, expanded form (/cygdrive/c/Users/Ives/Desktop/cyghome on my computer). Because of this, we're filtering it through sed and replacing this long string with ~ (note that because of the way sed works, this might not work if you have characters except for letters, digits and forward slashes in your home directory).

This should work on recent versions of bash and on most terminal emulators (this includes PuTTY and the Cygwin bash.exe).

Three Ways of Proving 0.999... = 1

More often than not, people don't believe me when I tell them that 0.99999... (followed by an infinite number of 9's) exactly equals 1. Over the years, I've accumulated 3 different ways to convince people this is the case.

Two of these are pretty much exactly the same, only the first is much more intuitive than the second. I can hear the skeptics yelling "Bring it on!", so here it goes.

The First Way Of Proving 0.999... = 1

Everybody knows that 1/3 = 0.333.... Just multiply both sides by 3 to get 3*1/3 = 3*0.333... which results in 1 = 0.999....

The Second Way Of Proving 0.999... = 1

Almost exactly the same as the first, only I'll be using a letter in this proof and people are afraid of letters when it comes to math. Intimidation is by far my favorite proof technique...

(1):  x = 0.999...
(2):  => 10x = 9.999...

Now (2) minus (1) gives 10x-x = 9.999... - 0.999..., so 9x = 9.000 and thus x = 1.

The Third Way of Proving 0.999... = 1

For the third proof, we'll need the formula for the sum of an infinite, converging, geometric series. Like Wikipedia says, the formula for this is (when -1 < r < 1)

S = \sum_{k=0}^\infty ar^k = \frac{a}{1-r}.

Now, let's rewrite 0.999...

0.999... = 0.9 + 0.09 + 0.009 + \cdots = {9 \over 10} + {9 \over 100} + {9 \over 1000} + \cdots = \sum_{n=1}^\infty \frac{9}{10^n}= \sum_{n=1}^\infty 9 \times \frac{1}{10^n}

Notice how the sum index in the last equation starts from 1 and the one in the Wikipedia formula starts from 0? No problem, we'll merely have to manufacture the first term out of thin air...

0.999... = \sum_{n=1}^\infty 9 \times \frac{1}{10^n} = 9 - 9 + \sum_{n=1}^\infty 9 \times \frac{1}{10^n} = 9 \times \frac{1}{10^0} - 9 + \sum_{n=1}^\infty 9 \times \frac{1}{10^n}= -9 + \sum_{n=0}^\infty 9 \times \frac{1}{10^n}

So a = 9 and r = 1/10 (which is between -1 and 1, so no problem there).

0.999 = -9 + \sum_{n=0}^\infty 9 \times \frac{1}{10^n} = -9 + \frac{a}{1-r} = -9 + \frac{9}{1-{1 \over 10}} = -9 + \frac{9}{9 \over 10} = -9 + 10 = 1

There are undoubtedly other ways to prove 1 = 0,999... but these three are my favorites and should be enough to convert even the strongest of unbelievers. Isn't math lovely?

Tips for editing Plain Text with Vim

Tip 1: Navigation

When writing Plain Text in Vim, navigation can be a bit tougher than it is while writing programming code. If you write in Vim like you would in Microsoft Word or in OpenOffice Writer, you are going to end up with incredibly long lines, since you probably won't be pressing ENTER at the end of every line.

Out of the box, the Vim j and k keys, change between actual lines (as indicated by line numbers), not between visual (wrapped) lines. If you would like to change between lines on the screen, gj and gk are your friend. You can even consider making this behavior the standard behavior of the j and k keys, by adding

map j gj
map k gk

to your .vimrc file (or by typing :map j gj and :map k gk in command mode when you start Vim).

Another possible solution to this problem (and the one I prefer), is to get Vim to place enters at the end of every screen-line. This is simply achieved by entering in command mode :set tw=78 (tw stands for text-width) if you'd like Vim to place a newline when a line of text is wider than 78 characters. 78 is a standard I like to use but you can of course choose your own value. Like always you can also place set tw=78 in your .vimrc or appropriate ftplugin file.

Even with text-width set, it is still very much possible to make a line either wider than you specified it should be or way too short, for example by joining multiple lines (SHIFT+J) or by deleting words. You could make the paragraph look good by manually placing hard returns and deleting them, or you could do it the smart way.

Moving newlines to make text fill the entered text-width is done with the gq command followed by a movement. To do this to your entire file (be careful, this is probably only appropriate if your file contains only text), you can enter the command gggqG. Explained:

gg          Go to the beginning of the file
  gq        Refill text from here to
    G       ... the end of the file

If you only want to refill a paragraph, probably a better idea than refilling an entire file, give the command gqap while anywhere in the paragraph. Once again explained:

gq          Refill text
  ap        in this paragraph (ap stands for "a paragraph")

Suppose your boss comes by after a few days and decides he wants the file you've been working on in Microsoft Word. Don't worry, you don't have to remove all of the newlines manually, there's a much better and faster way to do that. Just enter the following command (it's a bit complex so I won't blame you if you bookmark this page so you can copy paste it when you need it) :g/./,/^$/j then press enter twice. An explanation:

:g               A global command (affects the entire file)
  /./            search for a letter (so no empty lines)
     ,/^$/       and process everything until you find an empty line
          j      by joining them

In order for this to work on the last paragraph, your file has to end with an empty line (or the /^$/ part would not be matched).

Tip 2: Spelling

Everybody makes mistakes (especially me), and sometimes that includes spelling mistakes, especially when writing in a foreign language. Although the Vim spelling correction is by no means perfect, it's pretty good. As you probably know, spelling isn't turned on by default, so you have to turn it on. You can do this by executing the command :setlocal spell spelllang=en_us, of course replacing en_us with the appropriate locale. If you only want Spell checking while editing certain files, no problem. Just add autocmd BufRead *.txt setlocal spell spelllang=en_us to your .vimrc if you'd like to use US English spell checking for all txt files.

Try typing gibberish and see how Vim underlines the incorrect word with a red line. Move the cursor to the underlined word and press z= in command mode to see Vim's alternatives to the incorrect word.

When Vim incorrectly underlines a word, move the cursor over the word and type zg in command mode to permanently add it to the list of correct words for the language you currently have selected. If you prefer to only temporarily mark this word as correct you can give the zG command.

If you see a word that's not underlined but you'd like it to be underlined because it's a wrong (bad) word, you can permanently make it wrong by positioning the cursor over it and giving the zw command (or the zW command to temporarily mark it as wrong).

Tip 3: Word Count

People often like to know how many words or lines they've typed, either because they want to know how much progress they've made or because there is a limit on the amount of text they are allowed to write (e.g. an essay or article).

There are two ways to determine this, the wrong way and the right way. The wrong way would be to pass the file through an external program like wc (UNIX and Linux users can try this by typing :!wc % in Vim command mode). In all honesty, this way isn't so bad, except for the fact that it's not portable and not very Vim-y.

The right way to do this is to type g CTRL+G (press and release g and then press CTRL+G). For the file I'm writing right now this will output

Col 4 of 4; Line 104 of 104; Word 934 of 934; Byte 5350 of 5352 

That's about it for interesting stuff about editing plain text in Vim. There's not much to it really, just remember to save your work often.

Why the name?

Because the letter Psi is not only the most interesting, but also the most powerful letter you can find. I'd like to see you rule The Seven Seas with your letter.

You can also use it to express the Schrödinger equation and the generalized positional states of a qubit in a quantum computer (I totally looked that up on Wikipedia).

Ψ is the greatest letter around and best of all, it's mine!

"We will never ask you for your password"

Today, my Internet broke. More accurately, I lost the capability to visit websites. Every website I tried to visit redirected to 'http://redirect.telenet.be/something' (Telenet being my ISP). Mind you, this is hardly an exceptional situation. Telenet often redirects every website you request for reasons such as "You reached 90% of the volume you are allowed to download this month. Would you like to continue downloading or would you like to continue downloading?".

What's special about this time, is that instead of actually showing me the page on which you can choose to continue surfing, I got an Apache 500 error page with a large backtrace. Today might turn out to be fun after all.

This is obviously a problem I can't fix myself, hence I spend the next 15 minutes listening to crappy classical music while waiting for a line to open. I explain the problem and the guy at the other end tells me to call another number. Great.

Unlike the Tier 1 support guy, this guy sounds like he's actually doing something. I can hear him clicking and sighing in the background. "Hmm, that's a weird error code...", he mumbles. "Unfortunately, there's nothing I myself can do for you. I'll have to launch an official investigation and ehm, I'll need your password". "Which password?", I ask him. "The one you use to log onto [ISP site]". I give my phone the stink eye and ask "Are you serious?". "Yes", he tells me, "if you insist, I can also tell them to reset your password, but then the investigation will take longer and you'll also have a new password. Just to be clear, you don't have to give me your password, and you should also know you can change the password at any time." I take a deep sigh and spell out my password.

The support guy and I agree that with some luck, the problem should (temporarily) fix itself tonight, because the amount of traffic consumed will be reset at 00:00h, just like it has on the 16th of every month the last few years.

We hang up and I spend the next few hours anxiously chatting on Live Messenger (like I said before, only HTTP(S) traffic was blocked), hoping I won't have to spend the next week without Internet, waiting for some programmer to write a 2 line long fix. Luckily, the problem did indeed resolve itself a little over midnight. I'm just hoping they fix it before it occurs again.

Who would have guessed my first post would turn out to be an angry rant. I'm feeling like a real blogger already.