Monday, December 14, 2015

Oh! But how would I know without 'running' it, if its ok?

Have you come across your teammates, checking in scripts without running it, with syntax errors, malformed XML documents, broken Makefiles .... well, there is some hope, you could help (by sharing this link! :-) )

Mostly, if its code in a compiled language (like C), developers at least compile the code (might not test!) before checking into version control. But, in case of XML documents, scripts etc, the code gets checked in, and it will not error out until some test case is completely run, or in the nightly regressions!

Its not really difficult to do some basic validation of scripts, Makefiles, or even XML documents before a check-in. Most languages/tools have an option to do basic checks - syntax checks, dry run (without actually executing the commands), bytecode generation etc. Here are some, which I often use:

Make

make -n

Does a dry-run, shows which targets would get built on an actual run.

Perl

perl -c

Does a compilation and says if syntax is OK or not (there are caveats with BEGIN blocks, but at least there is something!)

Python

pycompile

This is good, there is a separate utility to do the compilation! (just run pycompile on your scripts once before you proceed)

bash

bash -n

Again, syntax checks without actually executing anything.

XML

xmllint

The name says it all! Even though there are plenty of options, you could just run once [without even options] on your XML docs to see if its well-formed.

Lua

luac

Lua bytecode compiler, like pycompile. Generates luac.out which you can delete :-)

Monday, December 07, 2015

Recover a Locked Android without data loss

Note3Recover
Since I had to recover a Samsung Note 3, all the steps mostly lean towards Samsung phones.

Scenario

  • Password forgotten! (you may think that this cannot happen to you, but it surely can! :-) )
  • Android Device Manager password reset doesn't work (and it didn't BTW! at least I could give this a try as WiFi was on)
  • Phone not registered with Samsung Recovery (good tha they have an alternate to ADM!, unlike ADM, you need to explicitly register your device - the one I was handling wasn't registered)
  • Un-rooted phone, with stock Android recovery
  • USB debugging disabled!

For the impatient

TL;DR summary
  • Install Samsung CDC drivers
  • Get the right device code (this one is ha3g)
  • Get the right custom recovery image (TWRP or CWM)
  • Use a firmware flashing utility which works for you (Odin or Heimdall)
  • Flash the recovery image.
  • Boot into recovery mode, and delete the password files.
Thats it!

Details

But,... the devil is in the details!

USB drivers

Get the Samsung CDC drivers and make sure you OS detects the phone. (If you are on Windows 10, and want to connect with ADB - Samsung doesnt have ADB drivers for Windows 10!)

Device Code Name

Now, this is one confusing thing! all the custom ROMs refer/name their images by phone's code.
The Samsung Note 3, for e.g. has multiple versions - the Sprint, Verizon and International. At first glance it might seem like "Ah! I bought it in India, and its not tied to any carrier, it must be international",... sorry! not that simple - you should know the correct CPU and Model (In this case the version is ha3g and the CPU is called Exynos though its not mentioned anywhere on the box or manuals! the model is N900 - this is easy should show up when the phone starts up)

Firmware Flash utility

2 choices here:
  • Heimdall (FOSS, available on Github. Binaries packages for both Windows and Linux available)
  • Odin (Leaked [from Samsung] Windows application)

Tryst with Heimdall

Since I have a love for FOSS, I was hell-bent to get Heimdall to work, and it had a cool command-line! I downloaded and built the latest version from source! (on Ubuntu 14.04). But, it didn't work! I could not figure out what the problem was! Tried Windows binaries too, and with USB2.0 and 3.0 port (with 3.0 port it would'nt even detect the phone!)
(the good part of trying Heimdall: I got to know a little bit about partitions and PIT [Partition Information Table])
A note on USB versions: If you do not know how to recognize the ports: peep into the USB socket, BLUE means 3.0 and YELLOW is 2.0.

Odin, finally

Odin is supposedly too picky: picky about the port, the cable etc. From what I read, if in case the phone is not detected on one port, try changing to another port and try using a different cable (in my case I had the original USB cable that came with the phone, and it was a USB3.0 - it got recognized instantly)
Odin's messages and the UI aren't too friendly either!

Custom recovery image

Here, again there are quite a few choices:
  • ClockWorkMod (CWM) - Development Ceased
  • Team Win Recovery Project (TWRP) New and cool
  • CyanogenMod Recovery (CR)
I could not find the CWM image for ha3g (though I could not find one for hltexx - not Exynos!). CR is still new! Initially, tried with whichever version of TWRP image that I could get, but Odin wouldn't flash it! It would give this error message:
NAND Write Start!! 
FAIL!
At first I thought it had something to do with the NAND flash storage! but after a lot more research found that it could be due to the type of image being written. I had to do 2 things to be able to successfully flash:
  1. Get the latest TWRP (2.8.6.0 recovery image, bundled as a .tar)
  2. Extract the .img from the .tar and convert it to a .tar.md5 (I found a script on XDA forums which did this)
And, finally Odin could flash the image to mobile!
You should put the mobile to Download Mode to write to flash, and that is achieved by pressing down Volume-Down, Home and Power keys together on Samsung

Recovery mode

You need to know how to get to recovery mode first, press down: Volume-Up + Home + Power keys and hold till logo flashes (do not confuse with Download-mode!)
The catch here is: though we flashed the TWRP recovery, the phone tries to be smart and replace with stock recovery if you let it reboot by itself! The remedy is to boot into TWRP immediately after the flash! i.e., once flashed do not let it restart normally (un-check Auto-Reboot in Odin)

Ah, TWRP!

Now, this is cool! if you ever have seen the default Android recovery and then compare with TWRP!, its like comparing age old feature phones to the moden day touch phone! TWRP has touch interface and neat buttons, you can pretty much do away without reading any manuals - thanks to the neat, and simplified UI.
Play safe
First thing I did after I could get to TWRP was to insert a microSD card, and take a backup of data, so that I could continue my RnD (it takes a NANDroid backup, which it can restore).
Recover/Remove?
I tried to pull out the locksettings.db and run some SQLite SQL queries to get the MD5 encoded password, and salt. At this stage, I didn't want to go any further. So, I came back to TWRP and deleted the 2 key files /data/system/password.key and /data/system/gesture.key (though I knew it was password-locked and not gesture locked). On reboot - No password! :) Nothing lost - all contacts and data intact.

Thanks

  • To this post for motivation! (so many Samsung service centers told me that its not possible, and that factory-reset [with data loss] is the only option!)
  • To this guy for the script
  • For many other detailed posts, step-wise procedures and YouTube videos which Android lovers have patiently put together.
  • And, of course to TWRP! I donated $15 towards the development (~1K/-)

Monday, November 30, 2015

Common config that can be utilized by multiple languages

Often, I would want to separate out some config/initial data out of a program, and keep it in a way that is accessible to different parts of the system. If all parts of the system are in the same language, then the config can be very simple - just the function invocation with values:

For e.g. in shell:

   # Format:
   # Employee name role department
   Employee "Bheem"  "Fighter"  "Security"
   Employee "Chutki" "Hiring"   "HR"

Then, the invoker would just have to define the function Employee and source this file!

So far, so good if all parts were in shell. But, what if I want to use this config in some C code, and in Lua scripts etc. ?

(Don't think file-read/parse/split/tokenize .. No no! yuck!)

Can we define (somehow), a common format which is usable by different environments (languages), with minimal or no effort!? Observe the calling conventions:

C function call looks like:

   Employee (arg1, arg2 /* ... */)

Shell (is the simplest):

     Employee "arg1" "arg2" # ...

Lua:

   Employee (arg1, arg2) -- ...

But, there is a special notation in Lua, when the argument is a string or a hash table - we can get rid of the parentheses! like:

   Employee "arg1"

OK, but what if we have more than 1 argument ? Now, we need to get a bit deeper into function-chaining:

    function Employee (name)
        return function (role)
            return function (department)
               add_to_db(name, role, department)
            end
        end
    end    

Without going into much details about anonymous functions, Lexical scoping, upvalues etc - the function returns a function, which returns yet another function - that means, when you invoke

    Employee "Bheem"  "Fighter"  "Security"

The first function reads the first value, and returns its inner function, which reads the second value and that in turn returns another function which finally does some work (and can access all the 3 values - this is what we want)! Cool, ain't it? :-)

Wait, we solved the problem for shell and Lua, what about C code ?

In case of C code, we can always use the well-documented Lua C APIs to read Lua config! :)
(after all, Lua evolved from data-description language)

If you think, all this is too much work - think again, know the DRY principle.

Tuesday, September 22, 2015

vim plugins that I cant live without!

Here is a list of vim plugins, which are absolute time-savers!

file_line

How many times, you'd have copied and pasted a line from gcc error/warning, in the form
/path/to/file.ext:line:col
and then had to remove the :line and :col before editing it in vim!
this nice little plugin will open into the same line and place the cursor at the right column!

matchit

For C, of course the built-in % key does the trick of taking you to the matching brace/paren, but what about languages where you have begin .. end, or if .. fi - that is where this is really useful - overloads the % key for keyword-pairs!

Align

Align anything! declarations, comments, function-headers ... saves a lot of time!

Fugitive

I use git, and no day goes by without looking into git blame of one or the other source file! Its neat to see that from within vim and can navigate to complete diff [if I have to] by just hitting enter!

DrawIt

I like to keep everything under version-control, so all my docs are in text (Markdown) and checked into git. I use DrawIt to create cool ASCII diagrams :)

Tuesday, September 08, 2015

Enforcing Coding-Style checks on diffs


Have you seen how most of the code review [comment]s look like?

- This line exceeds 80 columns ...
- No space after ...
- No comma here ...
- Conditionals must have a block ...

Though these look insane, they ensure that code looks saner, when everyone follows the company's coding guidelines - unfortunately its not easy to enforce! Everyone has their own favorite editor and their own settings, add to this - there will be third-party code which has its own different style.

The best one can do is - when new code is added, can warn if the delta violates coding guidelines or not. Wait, delta? - diff ? How do we run checks on a diff!? Its not easy, but not too complicated either! at least for most common checks which the author should have figured out himself/herself before posting the code for review!

My approach: Simple Perl regex checks on diff hunks!

(If the mention of Perl + regex makes you feel nauseatic - stop here. :-))

Note: I tried this for C code diffs only.

We're not dealing with a function in its entirety, its just a diff, so how do we go about ?
of course, line-by-line :-/ duh!

  • Use unified diff, to get the context (file name, line range) 
  • Do some basic line merging logic when we are sure its not complete [1]
  • Check line by line, for basic style enforcement
    • length > 80?
    • trailing white-space ?
    • if/else not followed by a block ({) ?
    • etc...
  • Keep track of line numbers so that meaningful messages can be printed, pointing to the exact line where the issue is!

And how do we run this script automatically ?
Since I use git, git hooks! (just make it part of pre-commit hook)

Does it work well?
surprisingly well! :-)

[1] I used parentheses balance as a check to know if I need to merge lines or not.


--EDIT--

Do we need to do any extra work, than what's written above?
We do have to take care of  /* comments */ , and "strings"! regexes can be easily fooled, if we don't take enough care!

Can we make this fool-proof?
No! this can only be best-effort