We consider totally irregular measures $\mu$ in $\mathbb{R}^{n+1}$, that is, $$\limsup_{r\to0}\frac{\mu(B(x,r))}{(2r)^n} >0 \;\; \& \;\; \liminf_{r\to0}\frac{\mu(B(x,r))}{(2r)^n}=0$$for $\mu$ almost every $x$. We will show that if $T_\mu f(x)=\int K(x,y)\,f(y)\,d\mu(y)$ is an operator whose kernel $K(\cdot,\cdot)$ is the gradient of the fundamental solution for a uniformly elliptic operator in divergence form associated with a matrix with H\"older continuous coefficients, then $T_\mu$ is not bounded in $L^2(\mu)$.This extends a celebrated result proved previously by Eiderman, Nazarov and Volberg for the $n$-dimensional Riesz transform and is part of the program to clarify the connection between rectifiability of sets/measures on $\mathbb{R}^{n+1}$ and boundedness of singular integrals there. Based on joint work with Mihalis Mourgoglou and Xavier Tolsa.