Gingrich says rivals' criticism taking a toll