FAQ/kappa/kappans

SPSS does not evaluate kappa for non-square tables where one of two raters does not use one or more ratings. The below syntax (downloaded from here) does this.

[COPY AND PASTE THE SYNTAX IN THE BOX BELOW AND ADJUST DATA INPUT AS REQUIRED]

* example data input template
*          
*                        Rater 2
*                  Mild Moderate Severe 
*         Mild      5      5       0
*Rater 1  Moderate  3      6       0
*         Severe    1      1       0 

set format f10.5.
data list free
/ r1 r2 freq.

begin data
1 1 5
2 1 3
3 1 1
1 2 5
2 2 6
3 2 1
end data.

*
* Syntax used for rectangular tables to compute kappa.
* (David Nichols, ASSESS Newsletter 1996)
* (recommended on P.104 of SPSS Reference manual, 1990):   
*
*
* Program uses Cohen's Kappa for agreement between a pair of raters
* for a two way rectangular table of ratings (ie at least 2 ratings given by both raters)
*
*
* Gives kappa and the asymptotic standard error of Everitt(1996)
* P.292 Making Sense of Statistics in Psychology
*
*
preserve.
set printback=off mprint=off.
save outfile='kap_0.sav'.
define kapparec (vars=!tokens(2) /num=!tokens(1) ).
count ms__=!vars !num (missing).
select if ms__=0.
matrix.
get x /var=!vars.
get ff /var=!num.
compute c=mmax(x).
compute y=make(c,2,0).
compute w=make(c,1,0).
compute sume=make(c,1,0).
compute ans=make(1,3,0).
loop i=1 to nrow(x).
loop k=1 to c.
do if x(i,1)=k.
compute y(k,1)=y(k,1)+ff(i,1).
end if.
do if x(i,2)=k.
compute y(k,2)=y(k,2)+ff(i,1).
end if.
do if (x(i,1) eq k and x(i,2) eq k).
compute w(k,1)=w(k,1)+ff(i,1).
end if.
end loop.
end loop.
loop k=1 to c.
compute sume(k,1)= y(k,1) * y(k,2)  / csum(ff).
end loop. 
compute kstat= ( csum(w) - csum(sume) ) / (csum(ff) - csum(sume)).
loop k=1 to c.
compute ans(1,1)=(csum(ff)-csum(sume)) / csum(ff).
compute ans(1,1)=ans(1,1)-(y(k,1)+y(k,2))*(csum(ff)-csum(w)) / (csum(ff))**2.
compute ans(1,1)=(w(k,1) / csum(ff))*ans(1,1)*ans(1,1).
compute ans(1,2)=ans(1,2)+ans(1,1).
end loop.
loop k=1 to c.
loop j=1 to c.
loop i=1 to nrow(x).
do if (x(i,1) eq k and x(i,2) eq j and x(i,1) ne x(i,2)).
compute ans(1,3)=ans(1,3)+ff(i,1)/csum(ff)*((y(k,2)/csum(ff))+(y(j,1)/csum(ff)))**(2).
end if.
end loop.
end loop.
end loop.
compute ans(1,3)=ans(1,3)*(1-(csum(w)/csum(ff)))**2.
compute ase=(csum(w)*csum(sume))/(csum(ff)*csum(ff)).
compute ase=ase-2*(csum(sume)/csum(ff))+(csum(w)/csum(ff)).
compute ase=ase**2.
compute ase=ans(1,3)-ase.
compute ase=ans(1,2)+ase.
compute ase=sqrt(ase*(1/(csum(ff)*(1-(csum(sume)/csum(ff)))**4))).
compute z=kstat/ase.
compute sig=1-chicdf(z**2,1).
save {kstat,ase,z,sig} /outfile='ka_tmp.sav'
     /variables=kstat,ase,z,sig.
end matrix.
get file='ka_tmp.sav'.
formats all (f11.8).
variable labels kstat 'Kappa' /ase 'ASE' /z 'Z-Value' /sig 'P-Value'.
report format=list automatic align(center)
  /variables=kstat ase z sig
  /title "Estimated Kappa, Asymptotic Standard Error,"
         "and Test of Null Hypothesis of 0 Population Value".
get file='kap_0.sav'.
!enddefine.

kapparec vars=r1 r2 num=freq.

The macro creates two new files, kap_0.sav and ka_tmp.sav, which contain respectively your original data, and a file used to store the values printed in the reported output.

MAKE SURE YOU HAVE NO EXISTING FILES CALLED KAP_0.SAV OR KA_TMP.SAV AS THESE WILL BE OVERWRITTEN.

Multiple invocations of the macro in the same interactive session will produce the following note:

>The macro name specified on the DEFINE command duplicates the name of a previously defined macro. This instance will take precedence.

This warning does not indicate a problem and may be ignored.