Hello, and welcome to FMH!
(a) [MATH]\left[\begin{array}{c}1 & 2 & -1 & -3 \\ 3 & 5 & k & -4 \\ 9 & k+13 & 6 & 9 \end{array}\right][/MATH]
(b)
−3R1+R2
[MATH]\left[\begin{array}{c}1 & 2 & -1 & -3 \\ 0 & -1 & k+3 & 5 \\ 9 & k+13 & 6 & 9 \end{array}\right][/MATH]
−R2
[MATH]\left[\begin{array}{c}1 & 2 & -1 & -3 \\ 0 & 1 & -k-3 & -5 \\ 9 & k+13 & 6 & 9 \end{array}\right][/MATH]
−9R1+R3
[MATH]\left[\begin{array}{c}1 & 2 & -1 & -3 \\ 0 & 1 & -k-3 & -5 \\ 0 & k-5 & 15 & 36 \end{array}\right][/MATH]
−(k−5)R2+R3
[MATH]\left[\begin{array}{c}1 & 2 & -1 & -3 \\ 0 & 1 & -k-3 & -5 \\ 0 & 0 & k^2-2k & 5k+11 \end{array}\right][/MATH]