I had a Toyota Matrix a while
I had a Toyota Matrix a while ago and it had the steering problem you described from day 1. Toyota technicians failed to find the cause of the problem even though they admitted that the steering was off. They balanced tires and adjusted wheels but the problem persisted.
When I had the brake pads replaced a few years later, the tech found that one of the brake pistons on the rear passenger side wheel was stuck. Replacing the piston fixed the weird steering problem, and it probably improved braking as well. I guess the brake pads on that wheel was lightly engaged from time to time because the piston failed to release the brake pad, which caused the steering problem.