Stack Overflows in SPARC

Causing and Thereby Avoiding Sparc stack overflows
--------------------------------------------------

Have been hitting my head against the screen to get an
understanding of the overflow problem on a sparc machine.
Ran into some pretty good examples but it took me some
time to get a good grip on the way to do this.

for the 32bit version of things
===============================
sample program from http://blogs.sun.com/peteh

    main()
        call x();
    x()
        call y();
    y()
        int a[1];
        a[20]=0

now why would this corrupt the stack and

    which stack and
    what part of the stack?

Let me try to answer these questions the best I can

    PSEUDO-CODE            ASSEMBLY
    main()               save %sp,...    %fp=%sp
        call x();        call x          %o7=pc
    x()                  save %sp,...    %fp=%sp
        call y();        call y          %o7=pc
    y()                  save %sp,...    %fp=%sp
        int a[1];
        a[20]=0

definition int a[1];
What makes this offset 20 special?
Look at the .s file it should show you an offset of the a[] array
in mine it was showing these two lines for a[20]=0

        add     %fp,-20,%l0
        st      %g0,[%l0+80]

Thus it shows that a is pointing to fp-20

                                        HigherAddr

fp of x()    ----------------------
                fr_argx[1]     _____________________________        
                fr_argd[6]    |nothing else coz there are no|
                fr_stret      |local variables etc in x()   |
frame.h         fr_savpc       -----------------------------    
from            fr_savfp       this is the registerw of main()
                fr_arg[6]
                fr_local[8]
fp of y()    ----------------------
fp-20           a   <---array a
             register window of x
             is saved here
current sp    ----------------------

                                        LowerAddr

so at fp you have fr_local        %l0-%l7
   at fp+8 you have fr_arg        %i0-i5
   at fp+14 you have fr_savfp     %i6
   at fp+15 you have fr_savpc     %i7    <-----return addr

so the return address offset from a[] is
        = +(20B+(15*4B))
        = +60B
        = +(20*4B)

so offset a[20] messes up the i7 for main and thereby the fault


now for the 64bit version of things
===================================
definition long a[1];
the assembly code shows

 or      %g0,1,%l3
        add     %fp,1991,%l1
        stx     %l3,[%l1+176]

so the array a is at fp+1991. Note the v9 arch specifies a 0x7ff stack bias
so this is within the stack even though you are adding to fp.

So where exactly is %o7 ?
Well 2047-1995 = 52B from the fp

fp+2047       -------------------

fp+1995        a


fp            -------------------

the return address offset from a[] is

 = +(52B+(15*8B))
        = +172B
        = +(22*8B)

so offset a[22] should corrupt i7 for main and cause a fault


So here we have managed to overwrite the %i7 register which stores
the return address. We have corrupted the stack of the x() function
so it cannot return correctly.




Sidenote: save instruction
Notice that most of the save instructions put in 96 bytes for the
stack. Why is that?

current sp      ---------------------------------
                    %i0-%i7 and %l0-%l7

                   (input and local vars)


sp+64           ---------------------------------   <- why 64? *A
                    hidden parameter
                ---------------------------------
                    %o0-%o5
               
                   (store first 6 params just in
                    case the params are passed by
                    addr and not by value)
sp+92           ---------------------------------
                   padding to be aligned
sp+96           ---------------------------------  <- why pad? *B

*A - why 64? - 8*2 registers = 8*2*(8bytes ie 4 locations) = 64
*B - why pad> - this is SPARC land
refer this link http://www.cs.unm.edu/%7Emaccabe/classes/341/labman/node12.html
thats all!

Technorati Tags: OpenSolaris, stackoverflow,sparc