Unexpected behavior when providing foreign key id instead of relationship when dealing with ORM factory

[paulshih85]

Description

I've created an ORM factory that generates a must-have relationship using SubFactory by default.
I expected it'll be replaced if I provide either the foreign key ID or the related object, but it didn't.

To Reproduce

There's a working example below.

Model / Factory code

# Model
class User(Base):
    __tablename__ = 'user'

    address_id = Column('address_id', Integer, ForeignKey('address.id', ondelete='CASCADE'), index=True, nullable=False)
    address = relationship('Address', cascade='all, delete')

# Factory
class AddressFactory(factory.Factory):

    class Meta:
        model = Address


class UserFactory(factory.Factory):

    class Meta:
        model = User

    address = factory.SubFactory(AddressFactory)

The issue

address = AddressFactory.build()
session.add(address)
session.commit()

user = UserFactory.build(address_id=address_id)

I expected that the created user's address would be the address I provided, but it seems that it is covered by the address created by the subfactory.