Sun GPL's Java

Well if you haven't heard this I guess you have been under a rock or are working at either Novell or Microsoft. Anyways Sun has just (as this is typed) released the entire Java source tree, atleast SE and ME, under the GPLv2 license.

I took a hard look at the comments on /. and frankly this is the first time that 90% of the comments seem to be positive. Now thats a major thumbs up, if releasing your IP as free software isn't enough to get your spirits high:)

Good luck Java.

Coolest Looking Sun Product

By now all of us have heard about Project BlackBox and what it actually is. I don't know the product details and the specs as such but what strikes me with this product is that Sun has finally managed to get in some good product design engineers in place. This is by far the coolest looking Sun product I have ever seen. The color combination, plexi and everything in there is not just functionally necessary but also aesthetically well placed. Good job!

Technorati Tags: sunmicrosystmes projectblackbox

Google testing OpenSolaris

There are rumors running all over the place that Google is Testing Sun's OpenSolaris. In fact even digg featured the article and the comments are quite interesting. Now leaving out the Linux fanatics who seem to be blindfolded in their love for their OS the reasonable amongst them can argue about the benefits about it. Now that would be a really fruitful discussion, which unfortunately never happens. Most threads of this nature are dominated by the "I am good and all else is bad" comments.

But taking a look at this from an unbiased perspective (I am really trying to be unbiased here) Google can seriously benefit from a lot of what OpenSolaris has to offer.

OpenSolaris is a BSD variant
All Unix and Linux faithful can't deny the fact that the BSD variants are for some reason the more sturdier class of *nixes. Be if FreeBSD, NetBSD, OpenBSD, Solaris of Apple's Darwin all of these have been very successful on high end systems. And high end today does not mean high cost but its more in terms of their scale. The up times shown by Solaris servers are legendary in their own right. And ofcourse now OpenSolaris is Trusted by default.

OpenSolaris has the bells and whistles
Dtrace, Zones, Zfs, FMA, x64 support and a lot more. Sun has gone all out while developing Solaris and has gone ahead and released all of this as OpenSolaris. Yes they are currently using the same code base for both, Solaris and OpenSolaris. So someone like Google can benefit by using zfs and scaling their mhamoth data centers even further while running dtrace to figure out the real time patterns in usage and code segments. So catch those algorithms while they are lazing around or eating more resources and improve them.

OpenSolaris provides for binary compatibility
Now I am sure Linux guys dint know what that means but it has been an assurance given by Sun on all Solaris systems for nearly 2 decades or more now. And hey its not just OpenSolaris that will let you maintain binary compatibility but because of the way the code is structured there should be no reason why this guarantee shouldn't be maintained between OpenSolaris variants as well.

OpenSolaris is Open
Ignore GPL devotees. If you don't like CDDL tough luck. But even with CDDL in place Google can go ahead and create a custom version of OpenSolaris for themselves and deploy it all over and heck even redistribute it if they like. Golaris sounds kiddish but well you never know:)

Suggestion to Google: Use Niagara ie USIV+ processors
The Niagara family of processors are targetted at customers such as Google that need a higher throughput on their front ends. With 8 cores and 4 threads per core in the Niagara1 family these processors can go ahead and eat up web queries on the Google website while the back end servers churn up the results. And NO the cost of the Niagara based systems is not prohibitive at all. In fact you could compare them with many desktop processors while comparing the cost/GHz and well the beat all of them in the cost/watt category. Am sure Google will be thinking about energy costs in the huge facility that they are supposedly coming up with in collaboration with the US government.

Well Netcraft shows gmail running some servers on Solaris 8 already. So I guess in a couple of months we should know if these rumors have any meat in them at all. And yeah being realistic even if Google adopts OpenSolaris it will be for very specific reasons. Their deployment of Linux is humongously distributed and very customized, from what I have heard. So we wont see them going on OpenSolaris 100% in the near future. And of course they too will have to douse off the in house Linux flame throwers too right. That may be the toughest challenge!

Well moving to OpenSolaris will simply mean that Google is definitely not resting on its laurels and is in constant search of whatever will let it improve further.


Technorati Tags: OpenSolaris Google Linux

Nevada on VMWare

It's really exciting to see OpenSolaris build 46 ie Nevada running smoothly on my laptop inside a VMWare workstation environment. I have dedicated 8GB and 512Mb RAM to its running instance and hopefully it will suffice. The main idea here is to keep up to speed with the new features being constantly integrated into Nevada while still doing my routine job. It was so much more fun while doing all this on the 7th floor of Divyasree chambers (Sun Microsystems IEC Bangalore)

Anyways the setup procedure in VMWare turned out to be pretty straightforward. The only hiccup was when selecting the disk for the installation. You actually need to uncheck and recheck the box again and only then things go on smoothly.

I have used VMWare a couple of times before and it seems to work alright most of the times. However there is a new kid on the block called Parallels and there were these rave reviews about it some time ago. Hope I can check that out sometime soon too.

For VMWare there is this wonderful set of slides on how to go about doing this right here. So there is no reason not to be running OpenSolaris even if you can't dedicate a complete box for it. Get onto the train now!

Technorati Tags: OpenSolaris VMWare

Postgresql Mod - Backup parts of a table

Well I got a chance to get into the postgres code although for a minor code addition. It turned out to be pretty cool just trying to understanding a small part of the source code.

Requirement.
1] Dump out only a part of a database table. eg oldest entries first

Approach
1] Now there are multiple ways to do it. For eg. you can go ahead and make a duplicate temp table with the rows that you want to dump out and then use the pg_dump tool to get you a copy as a binary file. Now that turns out to be a good idea if the amount of data to be moved around is relatively small as compared to the database, since the duplication will require a proportionate amount of space itself right.
2] The other way is to go ahead and modify the pg_dump tool itself to provide you with a partial dump.
Ofcourse keeping in with tradition we take the tougher approach ie the second one.


Once you get into the pg_dump source you realise that all it does is a dumb COPY command which does the real man's job of accessing the data for a given table/database and putting it out a given file. Onto the COPY source code in $PGSOURCE/src/backend/commands/copy.c

So what the COPY command does internally is to get to the database/table requested and redirect the binary file onto the specified file. Now if you want to dump out part of a table then you need some sort of query to be able to partition the part of the table that you desire. However doing this turns out to be very inefficienct since that would require you to have accesses into the table to get each row and then do comparisons on it and selectively dump to file. Not a good thing at all considering the large number of entries that we have.

Now since our requirement specifies that we need a part of the table to be dumped out, more specifically the older data we can work on the table if it has some timelined indices. So searching high and low in the docs I came across the funda of OIDs. Now OIDs or Object Identifiers are 4byte (on a 32bit m/c) integers that are unique per tuple on every table within the database. Moreover OIDs are accessed for dumps, it is a parameter to pg_dump, lucky us. Moreover since OIDs are appended to the tuples by the database itself on insertion we can rely on an effective technique being used in there. Something better than having triggers on each insert I hope!

So I go ahead and add two more parameters to the COPY command
1. from_oid
2. upto_oid
which gives us the range of OID's that we are interested in. We should be able to get these two values by checking the min and max values of the OIDs for a given table and simply do a percentage addition on the min value to get the upto_oid. You also need to make sure the parser recognises they new options so head off to $PGSOURCE/src/backend/parser/gram.y and good luck.

so now my customised postgres works as such
pg_dump from_oid 1634 upto_oid 1734
(internally)
COPY -t tablename TO filename WITH OIDS FROM_OID 1634 UPTO_OID 1734;


And if you really want to do some meaningful stuff. Take a look at this dude's webpage. Thanks Neil the tips were really helpful.

Technorati Tags: Postgresql mod oids pg_dump

cpp tasks in Apache ANT

I was recently introduced to 'ant'. A very simple make tool that allows for a generic way of carrying out your build process without too many hassels. Most of the work was done for having it as the build tool for java, however it has considerably evolved and is used all over the place, especially for cpp. This is the snippet used for getting the cpp make process done... way more simpler than the standard Makefile right.

<target name="compile-cpp" depends="init">
       <cc debug="false" link="executable" multithreaded="true" name="g++" outfile="${basedir}/bin" warnings="none" objdir="${basedir}/tmp">
           <fileset dir="${basedir}/src">
               <include name="*.cpp">
           </include>
               <compilerarg value="-Wall">
               <compilerarg value="-g">
               <compilerarg value="-c">
                   <linkerarg value="-lpthread">
               </linkerarg>
               </compilerarg>
               </compilerarg>
               </compilerarg>
           </fileset>
    </cc>
</target>

Technorati Tags: apache,ant,cpptask

DTrace on OSX

Well I haven't formatted my previous post correctly as yet but this news is just too good to keep out.

MacOSX Leopard will have inbuilt support for DTrace and that too within XCode.

Now for those of you who haven't used OSX, guys you haven't seen a true and complete OS yet, ah if only it used OpenSolaris as its choice of BSD platform it would be a marriage made in heaven, with Solaris powering the hardware while Aqua playing the sweet enchantress for the common user and ofcourse having the luxury of running the best and advanced version of Microsoft Office. And all that on the new and improved low-power Intel beasts. Come in VMWare and Parallels to create such sweet harmony that could make the tech gods dance.

But for now congrats to Team DTrace and the guys at Apple for recognising the beauty of the beast that is DTrace.

Tehcnorati Tags: OSX, Leopard,OpenSolaris, DTrace, XCode, MSOffice, Parallels, VMWare

Stack Overflows in SPARC

Causing and Thereby Avoiding Sparc stack overflows
--------------------------------------------------

Have been hitting my head against the screen to get an
understanding of the overflow problem on a sparc machine.
Ran into some pretty good examples but it took me some
time to get a good grip on the way to do this.

for the 32bit version of things
===============================
sample program from http://blogs.sun.com/peteh

    main()
        call x();
    x()
        call y();
    y()
        int a[1];
        a[20]=0

now why would this corrupt the stack and

    which stack and
    what part of the stack?

Let me try to answer these questions the best I can

    PSEUDO-CODE            ASSEMBLY
    main()               save %sp,...    %fp=%sp
        call x();        call x          %o7=pc
    x()                  save %sp,...    %fp=%sp
        call y();        call y          %o7=pc
    y()                  save %sp,...    %fp=%sp
        int a[1];
        a[20]=0

definition int a[1];
What makes this offset 20 special?
Look at the .s file it should show you an offset of the a[] array
in mine it was showing these two lines for a[20]=0

        add     %fp,-20,%l0
        st      %g0,[%l0+80]

Thus it shows that a is pointing to fp-20

                                        HigherAddr

fp of x()    ----------------------
                fr_argx[1]     _____________________________        
                fr_argd[6]    |nothing else coz there are no|
                fr_stret      |local variables etc in x()   |
frame.h         fr_savpc       -----------------------------    
from            fr_savfp       this is the registerw of main()
                fr_arg[6]
                fr_local[8]
fp of y()    ----------------------
fp-20           a   <---array a
             register window of x
             is saved here
current sp    ----------------------

                                        LowerAddr

so at fp you have fr_local        %l0-%l7
   at fp+8 you have fr_arg        %i0-i5
   at fp+14 you have fr_savfp     %i6
   at fp+15 you have fr_savpc     %i7    <-----return addr

so the return address offset from a[] is
        = +(20B+(15*4B))
        = +60B
        = +(20*4B)

so offset a[20] messes up the i7 for main and thereby the fault


now for the 64bit version of things
===================================
definition long a[1];
the assembly code shows

 or      %g0,1,%l3
        add     %fp,1991,%l1
        stx     %l3,[%l1+176]

so the array a is at fp+1991. Note the v9 arch specifies a 0x7ff stack bias
so this is within the stack even though you are adding to fp.

So where exactly is %o7 ?
Well 2047-1995 = 52B from the fp

fp+2047       -------------------

fp+1995        a


fp            -------------------

the return address offset from a[] is

 = +(52B+(15*8B))
        = +172B
        = +(22*8B)

so offset a[22] should corrupt i7 for main and cause a fault


So here we have managed to overwrite the %i7 register which stores
the return address. We have corrupted the stack of the x() function
so it cannot return correctly.




Sidenote: save instruction
Notice that most of the save instructions put in 96 bytes for the
stack. Why is that?

current sp      ---------------------------------
                    %i0-%i7 and %l0-%l7

                   (input and local vars)


sp+64           ---------------------------------   <- why 64? *A
                    hidden parameter
                ---------------------------------
                    %o0-%o5
               
                   (store first 6 params just in
                    case the params are passed by
                    addr and not by value)
sp+92           ---------------------------------
                   padding to be aligned
sp+96           ---------------------------------  <- why pad? *B

*A - why 64? - 8*2 registers = 8*2*(8bytes ie 4 locations) = 64
*B - why pad> - this is SPARC land
refer this link http://www.cs.unm.edu/%7Emaccabe/classes/341/labman/node12.html
thats all!

Technorati Tags: OpenSolaris, stackoverflow,sparc

Intro

I have been blogging since quite some time but most of it has been in relation to my country India and things that I encounter on a daily basis while I am here. As much as those issues are close to my heart professionally I am an engineer who has been working on Solaris for some time and the bond strengthened when I worked for Sun's Engineering Centre at Bangalore. I am a novice in front of the stalwarts at Sun but would like to put some things that I pick up out here. Personally I am not too much of a theoretical guy so these posts should be more practical pointers rather than technical revelations. As the url mentions am just a kid compared to the most out there so please feel welcome to point out the mistakes.