Oh! But how would I know without 'running' it, if its ok?

Have you come across your teammates, checking in scripts without running it, with syntax errors, malformed XML documents, broken Makefiles .... well, there is some hope, you could help (by sharing this link! :-) )

Mostly, if its code in a compiled language (like C), developers at least compile the code (might not test!) before checking into version control. But, in case of XML documents, scripts etc, the code gets checked in, and it will not error out until some test case is completely run, or in the nightly regressions!

Its not really difficult to do some basic validation of scripts, Makefiles, or even XML documents before a check-in. Most languages/tools have an option to do basic checks - syntax checks, dry run (without actually executing the commands), bytecode generation etc. Here are some, which I often use:

Make

make -n

Does a dry-run, shows which targets would get built on an actual run.

Perl

perl -c

Does a compilation and says if syntax is OK or not (there are caveats with BEGIN blocks, but at least there is something!)

Python

pycompile

This is good, there is a separate utility to do the compilation! (just run pycompile on your scripts once before you proceed)

bash

bash -n

Again, syntax checks without actually executing anything.

XML

xmllint

The name says it all! Even though there are plenty of options, you could just run once [without even options] on your XML docs to see if its well-formed.

Lua

luac

Lua bytecode compiler, like pycompile. Generates luac.out which you can delete :-)

Enforcing Coding-Style checks on diffs

Have you seen how most of the code review [comment]s look like?

- This line exceeds 80 columns ...
- No space after ...
- No comma here ...
- Conditionals must have a block ...

Though these look insane, they ensure that code looks saner, when everyone follows the company's coding guidelines - unfortunately its not easy to enforce! Everyone has their own favorite editor and their own settings, add to this - there will be third-party code which has its own different style.

The best one can do is - when new code is added, can warn if the delta violates coding guidelines or not. Wait, delta? - diff ? How do we run checks on a diff!? Its not easy, but not too complicated either! at least for most common checks which the author should have figured out himself/herself before posting the code for review!

My approach: Simple Perl regex checks on diff hunks!

(If the mention of Perl + regex makes you feel nauseatic - stop here. :-))

Note: I tried this for C code diffs only.

We're not dealing with a function in its entirety, its just a diff, so how do we go about ?
of course, line-by-line :-/ duh!

• Use unified diff, to get the context (file name, line range) 
• Do some basic line merging logic when we are sure its not complete [1]
• Check line by line, for basic style enforcement
• length > 80?
• trailing white-space ?
• if/else not followed by a block ({) ?
• etc...
• Keep track of line numbers so that meaningful messages can be printed, pointing to the exact line where the issue is!

And how do we run this script automatically ?
Since I use git, git hooks! (just make it part of pre-commit hook)

Does it work well?
surprisingly well! :-)

[1] I used parentheses balance as a check to know if I need to merge lines or not.


--EDIT--

Do we need to do any extra work, than what's written above?
We do have to take care of  /* comments */ , and "strings"! regexes can be easily fooled, if we don't take enough care!

Can we make this fool-proof?
No! this can only be best-effort

Merge Excel sheets ?

No problem!

A friend of mine had a problem, she had to merge two [huge] Excel workbooks, by matching names in one, with the names in another. Something like:

WorkBook1
+---------+---------+----------+----------+
| Name1   |  field1 |  field2  |  ...     |
+---------+---------+----------+----------+
|  ...    |         |          |          |
+---------+---------+----------+----------+
|  ...    |         |          |          |
+---------+---------+----------+----------+

WorkBook2
+---------+---------+----------+----------+
| field2_1| field2_2|  Name2   |  ...     |
+---------+---------+----------+----------+
|  ...    |         |          |          |
+---------+---------+----------+----------+
|  ...    |         |          |          |
+---------+---------+----------+----------+


If they were Database tables, then we could have done something on the lines of:

SELECT * FROM WorkBook1,WorkBook2 WHERE Name1=Name2;

But, these are excel sheets, and worse yet, the names in WorkBook1 are in format:
"FirstName LastName"
and names in WorkBook2 are in format:
"LASTNAME, FIRSTNAME"

(i.e, uppercase, and with comma) Duh! and there will be many names with 3-4 words - imagine the permutations.

Excel experts could say this can be solved with some cool macros, or maybe VB scripts!, but I am old-school Unix text guy!. I think in terms of text filters only!

To solve the problem of name matching, Take this [hypothetical] name:

Shankaran Ganesh Krishnan

the permutations will be:

Shankaran Krishnan Ganesh
Krishnan Shankaran Ganesh
Krishnan Ganesh Shankaran
Shankaran Ganesh Krishnan
Ganesh Shankaran Krishnan
Ganesh Krishnan Shankaran

Some names can also contain the initials [with a period], like:

Ganesh K. Shankaran

So, how can we do the name matching ? ... for a moment I thought of using a permuter
and then saving all permutations (stupid!), but that's not required!

Lets say we do the following,
 - Remove dots and commas
 - Change to lowercase (trim spaces too)
 - Sort the names by words

If you had "Shankaran, Ganesh Krishnan" in WorkBook1, and "GANESH, SHANKARAN KRISHNAN" in WorkBook2, then both will become: "ganesh krishnan shankaran"

Now, the only problem that remains, is to save the .xls as .csv, so that I can load it to Perl (Parse::CSV). Unfortunately, Excel doesn't have an option to save all the sheets in the workbook
at once to CSVs, I had to do that manually for each sheet and then merge. Other than that, its pretty straight forward.

If you are about to say: Show me teh codez!
here you go ...

What good is coding skills, if you cannot put it to use, at the right time, to help friends!? :-)

Colorizing GDB backtrace

Being in a startup, we get to see a lot of core dumps, every day ;-). With too many stack frames and long file+function names, I hate the output!. In fact the visual clutter is so much, that it takes me long time to compare two tracebacks! I was thinking what can be done about it - hit upon gdb hooks in a SO post. Put together with my all time fav - Perl, to color the output, got some nice colorful tracebacks! ;-). The following is what I put in my ~/.gdbinit

shell mkfifo /tmp/colorPipe.gdb
# no pause/automore 
set height 0
# dont break line
set width 0
define hook-backtrace
        shell cat /tmp/colorPipe.gdb |  ~/highlgdb.pl &
        set logging redirect on
        set logging on /tmp/colorPipe.gdb
end
define hookpost-backtrace
        set logging off
        set logging redirect off
        shell sleep 0.1
end
define hook-quit
        shell rm /tmp/colorPipe.gdb
end
define re
   echo \033[0m
end

And the Perl script (highlgdb.pl) to do the highlight:

Try it!, at least identifying the culprit will be much quicker, if not fixing the issue! ;-)

The FizzBuzz of scripting

I have read a lot of articles on codinghorror, joelonsoftware and the like, about the FizzBuzz test, but somehow when I have to actually take a F2F interview, and the candidate is like 6-8+ years experienced, I'll have some hesitation to actually start with FizzBuzz. Instead, I start with a casual conversation about the past work s/he has done and see if I can probe more on any of the keywords from their resume. Recently, I had to take an interview of an engineer for QA role, and all that I was asked to check was How good is he in Perl, now, I don't want to get into a debate on Perl, its TIMTOWTDI philosophy etc. Somehow after the first few minutes of talking, this idea struck me to ask him about a variant of FizzBuzz - I will term it as The FizzBuzz of scripting, i.e,

Split an input [text] file on the following criteria:
if line matches foo write it out to /tmp/foo.out
if line matches bar write it out to /tmp/bar.out
if line matches foo and bar write it out to /tmp/foobar.out

The candidate started out like: "I can use multiple greps, and write a simple shell script..". Then I added a little more constraints: has to be fast, input is large, and I want a more

robust and maintainable script.

The initial Perl script he came up with, using regex was fine, then I wanted a little more tweaking, so I started asking about the points in code, where optimization was possible, at some point he gave up that no optimization was possible.

All I was looking for, was the simplest change - that you can compile the regex once and then match many-times over!

/.*mypattern$/ and print; # Does a compile and match

This will not be obvious to many programmers who haven't tried different flavors of regexes and in different languages, that, applying a regex is a two stage process:

1. Compile
2. Match

$re = qr/.*mypattern$/; # Compile..
/$re/ and print; # Match 

Usually the compile is compute intensive, and if the pattern isn't going to change, then it’s best to compile once and store. Some languages clearly separate the steps, like

re.compile ()
..
re.match()

Of course this is how we would do it in C using POSIX regex as well

regcomp()
..
regexec()

The point I am trying to make, is that certain things are not obvious. Being polyglot'y helps, if not, a lot of reading. Maybe I am belabouring the obvious, but sometimes the obvious isn't too obvious! :)

Edit/open files quick

As developers, we edit lot of source files, and each has a specific extension
and mostly we either type them or let the shell do the auto-completion.
Most of the source files that I work on, follow this long_underscore_separated_names convention, and I hate to even press TAB TAB TAB...
When I know a file name, or am seeing it on console (with the previous ls), I want to be able to open a file (in the current directory) with as few keystrokes as possible.

I came up with a small Perl script, which, when combined with shell aliases
can be a real time saver (and save you from RSI as well! :) )
The same effect can be achieved with sed/awk or any other regex-well-supported
languages, but Perl is my favorite.

Approach

Given a file like: my_lisp_parser.c,  I want to be able to call it as
m l p or just mlp (first letters of the words, assuming file names separated with
underscore, or hyphen)
The extensions, lets not worry for now...
So, how do we go about?
The regex way...
build a regex from these characters and then do a quick look-up in the current dir:
like, m_l_p.
How to build this regex:
simple, split the input if its one word, or concat if its multiple words, but
before joining, add [^_-]*, that's the equivalent of our something above.

What to do in case of ambiguity, i.e, if we have a file like
my_lex_print.c (some hypothetical name, duh!)
then, using mlp will have a problem, so we should say
m le p
where le is the disambiguator.

So, now we can edit some files, with some extensions, what about those with
no extension ? usually I edit a lot of Makefile, that, I can handle as a
special case.

The extension is given as an argument to the script, but while using this
script, we do not invoke it directly, but with shell aliases, therefore
typing the extension will not be necessary. I use bash BTW, so here are my bash aliases (for most commonly used file extensions)

alias vc='~/short_open.pl c'

alias vh='~/short_open.pl h'

alias vp='~/short_open.pl p[ly]'

alias vy='~/short_open.pl yml'

alias vd='~/short_open.pl diff'

alias vs='~/short_open.pl sh'

alias vl='~/short_open.pl l[ou][ga]'

alias vm='~/short_open.pl mk'

I can open a .c file like vc mlp, or a .h file like vh mlp

The full script is here:

$editor = shift;

$ext = shift;

if ( scalar(@ARGV) == 1 ) {

    $in = shift;

    if ((length($in) == 1) and ($in eq 'm')) {

        system("$editor [Mm]akefile");

        exit 0;

    }

    if ((length($in) == 1) and ($in eq 'R')) {

        system("$editor README");

        exit 0;

    }

    if ((length($in) == 1) and ($in eq 'w')) {

        system("$editor wscript");

        exit 0;

    }

    @p = split( //, $in ); # 1 arg - pick and split each char

}

else {

    @p = (@ARGV); # multi arg - use them as is

}

$res = '^';

$res .= join( '[^_-]*[-_]', @p );

$res .= '[^_-]*[.]';

$res .= $ext . '$'; 

$re = qr/$res/;

foreach $a ( glob("*.$ext") ) {

    if ( $a =~ m/$re/ ) {

        print " [$a] \n";

        system ("$editor $a");

        exit 0;

    }

}

# If it were not really a long name with underscore, then try

# a plain short name (assume 1 arg)

if ( scalar(@ARGV) > 1 ) {

    exit 1;

}

$res = $in;  

$res .= '.*[.]';  # deliberate unbounded match so that any part

$res .= $ext;     # of the file name can be input

$re = qr/$res/;

foreach $a ( glob("*.$ext") ) {

    if ( $a =~ m/$re/ ) {

        print " [$a] \n";

        system ("$editor $a");

        exit 0;

    }

}