TIFU by realizing I'm perpetuating my boyfriend's trauma

I would like to start this post by saying I know that what happened to my boyfriend wasn't my fault, Im not implying that, but I know I struggle with articulation so I wanted to clear that up. Obvious throwaway, and apologies for formatting.

My boyfriend (20 M) and I (21 TF) have been since together since October, and not too long before we started dating he was raped. He's mentioned to me that he may have problems with intimacy, and that sex was off the table in any relationship before we started dating, and I was fine with that. I'm a very cuddly person, and he knows that. He's told me in the past he wants to heal and wants me to be intimate, but I'm now realizing the extent to what this is doing to him. We were cuddling in my car and he goes silent, not answering me verbally, which is a sign he's kinda out of it (we're both autistic and go non verbal when overstimulated, so it is my sign something is wrong) and turns out he's having a panic attack. The only other time I know of this happening to him was when we were making out and he noticed I was aroused. After I did my best to help him snap out of it, which I only hope I did well as I'm not used to situations like this, he told me something that I haven't stopped thinking about. "It just feels weird when I'm being held, and when I think about why I'm suddenly back to that place" (not going into detail here).

I'm a pretty insecure person, and I always have this insecurity about being weird, creepy, pervvy, etc. Being a trans woman I've kinda lived with the stereotype that I'm a predator. Because of this I've been really careful with my boyfriend, when we first started dating I'd ask for consent for everything, holding hands, cheek kisses, etc. Eventually he said i didn't need to, and he'd tell me if something was up, and he didn't want me feeling bad or hesitant to be intimate because of what happened to him because that'd only make him feel bad. So I trusted that, but now he's telling me something new about our intimacy and I don't know how to feel. I was terrified of doing something and hurting him, and I know I wouldn't do anything intentionally, so I've been trying my best to notice if I was doing something wrong. The first time he had a panic attack with me he said something felt different, the idea that someone was physically attracted to him kinda brought him back to that moment he was raped, and although I felt guilty, I pushed it down, saying it was irrational and put my full faith in him.

However now I can't stop thinking about that, this whole time we've been dating have I been doing this? Because he's has moments of being nonverbal before, but there's a difference between that and a panic attack, and up until now I assumed he'd just been zoned out. He zones out a lot when I'm driving him places and stuff so I thought it was that, but now I'm double guessing myself. He's aware of my insecurity, and now I'm wondering if he's been holding back his uncomfortable feelings to make sure I didn't feel guilty. I want to ask him if it's something he goes through more often than I realize, but I don't know if I could ever live with myself if he says yes.

TL;DR: my intimacy with my boyfriend caused a panic attack and now I'm questioning my actions through our relationship.

(Also apologizes if this breaks the "has to be your fault" rule, but I think this is more than just being insecure at this point)