I assume it's a single engine.....
Mine does exactly the same thing, pulls to port. I cured mine by fitting auto helm. Before fitting the AH I had a piece of string that went over the screen wash knob and around the wheel handle, that gave me a little time in calm conditions to at least put the kettle on before going off course.
Don't worry about it
Born to cruise...........once forced to work!