-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Temporal table migration regression from EF Core 8 to 9 #35108
Comments
I have the same problem. |
@roji I can see you have changed the title to only indicate that it's temporal table related, but the second issue in my post does not seem related at all to temporal tables, but rowversion and it's default value. Maybe I'm wrong tho and they are both related to temporal, just wanted to make sure that part doesn't get overlooked. |
@Moerup I indeed missed the 2nd issue - but it would be better to have that as a separate issue, as it indeed seems unrelated. |
I get same error as @bhp15973 Resulting in this error from SQL: As far as I can tell this is a standard migration of a non-temporal table to a temporal table. This migration works in efcore 8. I believe the issue is the ADD COLUMN and PERIOD FOR SYSTEM_TIME need to all happen at the same time: ALTER TABLE dbo.YourTable in EF8 "GENERATED ALWAYS" is not specified. |
I have created a repo that presents a minimal reproduction of what I think is the same or similar issue as OP. @Moerup let me know if this is a different issue and I will create a new issue. This is a blocking issue for my team adopting EFCore 9, which we really want to do! https://github.com/danroot/Spike.EFCore9TemporalTable Steps to reproduce with this repo:
Steps to reproduce from scratch:
|
We're also having the exact same problem with a migration that converts a non temporal table into a temporal table. As we apply migrations from code when developing, this causes issues if there is no database available yet. The problem is that Aspire 9 is based on EFCore 9 so we can't upgrade Aspire until this is fixed but Aspire only supports the last version so until this is fixed we're using a version of Aspire that is out of support. So we hope this gets fixed soon. |
@WolfspiritM You can use Aspire with EFCore 8. In our main solution, because of this regression, we have Aspire running an ASP.NET Core 9 project using EFCore 8. The main thing I think might not work is health checks against the ef context. If you want a containerized SQL in development:
If you don't want containerized SQL Server, just inject the connectionstring:
In the api, the implication is that instead of this:
We just continue to do this, same as always:
Of course we do want to get to EFCore 9, but hopefully this makes it less of a block. In our case, the impact is that we can't run integration tests under EFCore 9. We have tests that create a blank database by running migrations, create a snapshot, then test and rollback snapshot after each test. In this scenario, the migrations to create the database fail. We also have code to let developers get a 'fresh' dev database from scratch. Again, the migrations fail in EFCore9, but not 8. A nuclear option that might work depending on the use of the migrations is to reset migrations. |
Thanks. My main issue was that the Aspire components for sqlserver were using efcore 9 packages. So I will give Aspire 9 with net8 a try now until this issue is resolved. |
getting same issue on migrating EF Core 8 to 9. we have some temporal tables and after updating efcore from 8 to 9. we are unable to update database through migration and getting following error.
|
I'm running into the same issue at the moment, also moving from 8 to 9. Similar error messages to others
|
In 9 we changed the way we process migration of temporal tables. One of the changes was drastically reducing the number of annotations for columns which are part of temporal tables. This however caused regressions for cases where migration code was created using EF8 (and containing those legacy annotations) but then executed using EF9 tooling. Specifically, extra annotations were generating a number of superfluous Alter Column operations (which were only modifying those annotations). In EF8 we had logic to weed out those operations, but it was removed in EF9. Fix is to remove all the legacy annotations on column operations before we start processing them. We no longer rely on them, but rather use annotations on Table operations and/or relational model. The only exception is CreateColumnOperation, so for it we convert old annotations to TemporalIsPeriodStartColumn and TemporalIsPeriodEndColumn where appropriate. Also, we are bringing back logic from EF8 which removed unnecessary AlterColumnOperations if the old and new columns are the same after the legacy temporal annotations have been removed. Also fixes #35148 which was the same underlying issue.
In 9 we changed the way we process migration of temporal tables. One of the changes was drastically reducing the number of annotations for columns which are part of temporal tables. This however caused regressions for cases where migration code was created using EF8 (and containing those legacy annotations) but then executed using EF9 tooling. Specifically, extra annotations were generating a number of superfluous Alter Column operations (which were only modifying those annotations). In EF8 we had logic to weed out those operations, but it was removed in EF9. Fix is to remove all the legacy annotations on column operations before we start processing them. We no longer rely on them, but rather use annotations on Table operations and/or relational model. The only exception is CreateColumnOperation, so for it we convert old annotations to TemporalIsPeriodStartColumn and TemporalIsPeriodEndColumn where appropriate. Also, we are bringing back logic from EF8 which removed unnecessary AlterColumnOperations if the old and new columns are the same after the legacy temporal annotations have been removed. Fixes #35108 Also fixes #35148 which was the same underlying issue.
thanks @danroot for the great repros! It was very helpful and fix is on the way. Several people are hitting this so we will try to fit this into the next available patch release (likely 9.0.2) In the meantime, here is the workaround. It's a bit of work and may not be realistic if affected migrations are large enough, but hopefully will unblock some folks. For the affected migration code (that work on temporal tables or convert from regular to temporal) look for
For AlterColumnOperation, remove all those annotations (as well as .OldAnnotations entries for those 5 values)
After removing those annotations, if there are no other changes in that operation (i.e. or of the "old" properties have the same value as the new ones) - remove the For CreateColumn, also remove those 5 annotations, but additionally for the period start column add:
and for period end column add:
If the migration creates a temporal table right away (rather than converting regular table into temporal), you also need to fix annotations on the CreateTable columns: this is how the operation should look like initially: migrationBuilder.CreateTable(
name: "Customers",
columns: table => new
{
Id = table.Column<int>(type: "int", nullable: false)
.Annotation("SqlServer:Identity", "1, 1")
.Annotation("SqlServer:IsTemporal", true)
.Annotation("SqlServer:TemporalHistoryTableName", "CustomersHistory")
.Annotation("SqlServer:TemporalHistoryTableSchema", null)
.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart"),
Name = table.Column<string>(type: "nvarchar(max)", nullable: false)
.Annotation("SqlServer:IsTemporal", true)
.Annotation("SqlServer:TemporalHistoryTableName", "CustomersHistory")
.Annotation("SqlServer:TemporalHistoryTableSchema", null)
.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart"),
PeriodEnd = table.Column<DateTime>(type: "datetime2", nullable: false)
.Annotation("SqlServer:IsTemporal", true)
.Annotation("SqlServer:TemporalHistoryTableName", "CustomersHistory")
.Annotation("SqlServer:TemporalHistoryTableSchema", null)
.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart"),
PeriodStart = table.Column<DateTime>(type: "datetime2", nullable: false)
.Annotation("SqlServer:IsTemporal", true)
.Annotation("SqlServer:TemporalHistoryTableName", "CustomersHistory")
.Annotation("SqlServer:TemporalHistoryTableSchema", null)
.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart")
},
constraints: table =>
{
table.PrimaryKey("PK_Customers", x => x.Id);
})
.Annotation("SqlServer:IsTemporal", true)
.Annotation("SqlServer:TemporalHistoryTableName", "CustomersHistory")
.Annotation("SqlServer:TemporalHistoryTableSchema", null!)
.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart"); this is what it should be changed to: migrationBuilder.CreateTable(
name: "Customers",
columns: table => new
{
Id = table.Column<int>(type: "int", nullable: false)
.Annotation("SqlServer:Identity", "1, 1"),
Name = table.Column<string>(type: "nvarchar(max)", nullable: false),
PeriodEnd = table.Column<DateTime>(type: "datetime2", nullable: false)
.Annotation("SqlServer:TemporalIsPeriodEndColumn", true),
PeriodStart = table.Column<DateTime>(type: "datetime2", nullable: false)
.Annotation("SqlServer:TemporalIsPeriodStartColumn", true)
},
constraints: table =>
{
table.PrimaryKey("PK_Customers", x => x.Id);
})
.Annotation("SqlServer:IsTemporal", true)
.Annotation("SqlServer:TemporalHistoryTableName", "CustomersHistory")
.Annotation("SqlServer:TemporalHistoryTableSchema", null!)
.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart"); do not remove annotations on the CreateTable operation itself, just on the columns Also note that period start and end columns each have an annotation added just like |
as an example, Up method from @danroot 's repro migration in the file 20241121163049_Temporal.cs should look like this: protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AlterTable(
name: "Persons")
.Annotation("SqlServer:IsTemporal", true)
.Annotation("SqlServer:TemporalHistoryTableName", "PersonsHistory")
.Annotation("SqlServer:TemporalHistoryTableSchema", null)
.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
migrationBuilder.AlterTable(
name: "Contacts")
.Annotation("SqlServer:IsTemporal", true)
.Annotation("SqlServer:TemporalHistoryTableName", "ContactsHistory")
.Annotation("SqlServer:TemporalHistoryTableSchema", null)
.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
//migrationBuilder.AlterColumn<string>(
// name: "LastName",
// table: "Persons",
// type: "nvarchar(100)",
// maxLength: 100,
// nullable: false,
// oldClrType: typeof(string),
// oldType: "nvarchar(100)",
// oldMaxLength: 100)
// //.Annotation("SqlServer:IsTemporal", true)
// //.Annotation("SqlServer:TemporalHistoryTableName", "PersonsHistory")
// //.Annotation("SqlServer:TemporalHistoryTableSchema", null)
// //.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
// //.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
//migrationBuilder.AlterColumn<string>(
// name: "FirstName",
// table: "Persons",
// type: "nvarchar(100)",
// maxLength: 100,
// nullable: false,
// oldClrType: typeof(string),
// oldType: "nvarchar(100)",
// oldMaxLength: 100)
// //.Annotation("SqlServer:IsTemporal", true)
// //.Annotation("SqlServer:TemporalHistoryTableName", "PersonsHistory")
// //.Annotation("SqlServer:TemporalHistoryTableSchema", null)
// //.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
// //.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
//migrationBuilder.AlterColumn<int>(
// name: "Id",
// table: "Persons",
// type: "int",
// nullable: false,
// oldClrType: typeof(int),
// oldType: "int")
// .Annotation("SqlServer:Identity", "1, 1")
// //.Annotation("SqlServer:IsTemporal", true)
// //.Annotation("SqlServer:TemporalHistoryTableName", "PersonsHistory")
// //.Annotation("SqlServer:TemporalHistoryTableSchema", null)
// //.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
// //.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart")
// .OldAnnotation("SqlServer:Identity", "1, 1");
migrationBuilder.AddColumn<DateTime>(
name: "PeriodEnd",
table: "Persons",
type: "datetime2",
nullable: false,
defaultValue: new DateTime(1, 1, 1, 0, 0, 0, 0, DateTimeKind.Unspecified))
//.Annotation("SqlServer:IsTemporal", true)
//.Annotation("SqlServer:TemporalHistoryTableName", "PersonsHistory")
//.Annotation("SqlServer:TemporalHistoryTableSchema", null)
//.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
//.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
.Annotation("SqlServer:TemporalIsPeriodEndColumn", true);
migrationBuilder.AddColumn<DateTime>(
name: "PeriodStart",
table: "Persons",
type: "datetime2",
nullable: false,
defaultValue: new DateTime(1, 1, 1, 0, 0, 0, 0, DateTimeKind.Unspecified))
//.Annotation("SqlServer:IsTemporal", true)
//.Annotation("SqlServer:TemporalHistoryTableName", "PersonsHistory")
//.Annotation("SqlServer:TemporalHistoryTableSchema", null)
//.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
//.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
.Annotation("SqlServer:TemporalIsPeriodStartColumn", true);
//migrationBuilder.AlterColumn<string>(
// name: "Phone",
// table: "Contacts",
// type: "nvarchar(20)",
// maxLength: 20,
// nullable: false,
// oldClrType: typeof(string),
// oldType: "nvarchar(20)",
// oldMaxLength: 20)
// //.Annotation("SqlServer:IsTemporal", true)
// //.Annotation("SqlServer:TemporalHistoryTableName", "ContactsHistory")
// //.Annotation("SqlServer:TemporalHistoryTableSchema", null)
// //.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
// //.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
//migrationBuilder.AlterColumn<int>(
// name: "PersonId",
// table: "Contacts",
// type: "int",
// nullable: true,
// oldClrType: typeof(int),
// oldType: "int",
// oldNullable: true)
// //.Annotation("SqlServer:IsTemporal", true)
// //.Annotation("SqlServer:TemporalHistoryTableName", "ContactsHistory")
// //.Annotation("SqlServer:TemporalHistoryTableSchema", null)
// //.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
// //.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
//migrationBuilder.AlterColumn<string>(
// name: "Name",
// table: "Contacts",
// type: "nvarchar(100)",
// maxLength: 100,
// nullable: false,
// oldClrType: typeof(string),
// oldType: "nvarchar(100)",
// oldMaxLength: 100)
// //.Annotation("SqlServer:IsTemporal", true)
// //.Annotation("SqlServer:TemporalHistoryTableName", "ContactsHistory")
// //.Annotation("SqlServer:TemporalHistoryTableSchema", null)
// //.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
// //.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
//migrationBuilder.AlterColumn<string>(
// name: "Email",
// table: "Contacts",
// type: "nvarchar(100)",
// maxLength: 100,
// nullable: false,
// oldClrType: typeof(string),
// oldType: "nvarchar(100)",
// oldMaxLength: 100)
// //.Annotation("SqlServer:IsTemporal", true)
// //.Annotation("SqlServer:TemporalHistoryTableName", "ContactsHistory")
// //.Annotation("SqlServer:TemporalHistoryTableSchema", null)
// //.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
// //.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
//migrationBuilder.AlterColumn<int>(
// name: "Id",
// table: "Contacts",
// type: "int",
// nullable: false,
// oldClrType: typeof(int),
// oldType: "int")
// .Annotation("SqlServer:Identity", "1, 1")
// //.Annotation("SqlServer:IsTemporal", true)
// //.Annotation("SqlServer:TemporalHistoryTableName", "ContactsHistory")
// //.Annotation("SqlServer:TemporalHistoryTableSchema", null)
// //.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
// //.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart")
// .OldAnnotation("SqlServer:Identity", "1, 1");
migrationBuilder.AddColumn<DateTime>(
name: "PeriodEnd",
table: "Contacts",
type: "datetime2",
nullable: false,
defaultValue: new DateTime(1, 1, 1, 0, 0, 0, 0, DateTimeKind.Unspecified))
//.Annotation("SqlServer:IsTemporal", true)
//.Annotation("SqlServer:TemporalHistoryTableName", "ContactsHistory")
//.Annotation("SqlServer:TemporalHistoryTableSchema", null)
//.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
//.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
.Annotation("SqlServer:TemporalIsPeriodEndColumn", true);
migrationBuilder.AddColumn<DateTime>(
name: "PeriodStart",
table: "Contacts",
type: "datetime2",
nullable: false,
defaultValue: new DateTime(1, 1, 1, 0, 0, 0, 0, DateTimeKind.Unspecified))
//.Annotation("SqlServer:IsTemporal", true)
//.Annotation("SqlServer:TemporalHistoryTableName", "ContactsHistory")
//.Annotation("SqlServer:TemporalHistoryTableSchema", null)
//.Annotation("SqlServer:TemporalPeriodEndColumnName", "PeriodEnd")
//.Annotation("SqlServer:TemporalPeriodStartColumnName", "PeriodStart");
.Annotation("SqlServer:TemporalIsPeriodEndColumn", true);
} |
In 9 we changed the way we process migration of temporal tables. One of the changes was drastically reducing the number of annotations for columns which are part of temporal tables. This however caused regressions for cases where migration code was created using EF8 (and containing those legacy annotations) but then executed using EF9 tooling. Specifically, extra annotations were generating a number of superfluous Alter Column operations (which were only modifying those annotations). In EF8 we had logic to weed out those operations, but it was removed in EF9. Fix is to remove all the legacy annotations on column operations before we start processing them. We no longer rely on them, but rather use annotations on Table operations and/or relational model. The only exception is CreateColumnOperation, so for it we convert old annotations to TemporalIsPeriodStartColumn and TemporalIsPeriodEndColumn where appropriate. Also, we are bringing back logic from EF8 which removed unnecessary AlterColumnOperations if the old and new columns are the same after the legacy temporal annotations have been removed. Fixes #35108 Also fixes #35148 which was the same underlying issue.
reopening for potential patch |
In 9 we changed the way we process migration of temporal tables. One of the changes was drastically reducing the number of annotations for columns which are part of temporal tables. This however caused regressions for cases where migration code was created using EF8 (and containing those legacy annotations) but then executed using EF9 tooling. Specifically, extra annotations were generating a number of superfluous Alter Column operations (which were only modifying those annotations). In EF8 we had logic to weed out those operations, but it was removed in EF9. Fix is to remove all the legacy annotations on column operations before we start processing them. We no longer rely on them, but rather use annotations on Table operations and/or relational model. The only exception is CreateColumnOperation, so for it we convert old annotations to TemporalIsPeriodStartColumn and TemporalIsPeriodEndColumn where appropriate. Also, we are bringing back logic from EF8 which removed unnecessary AlterColumnOperations if the old and new columns are the same after the legacy temporal annotations have been removed.
… 9 (#35289) In 9 we changed the way we process migration of temporal tables. One of the changes was drastically reducing the number of annotations for columns which are part of temporal tables. This however caused regressions for cases where migration code was created using EF8 (and containing those legacy annotations) but then executed using EF9 tooling. Specifically, extra annotations were generating a number of superfluous Alter Column operations (which were only modifying those annotations). In EF8 we had logic to weed out those operations, but it was removed in EF9. Fix is to remove all the legacy annotations on column operations before we start processing them. We no longer rely on them, but rather use annotations on Table operations and/or relational model. The only exception is CreateColumnOperation, so for it we convert old annotations to TemporalIsPeriodStartColumn and TemporalIsPeriodEndColumn where appropriate. Also, we are bringing back logic from EF8 which removed unnecessary AlterColumnOperations if the old and new columns are the same after the legacy temporal annotations have been removed.
fixed for 9.0.2, closing |
Migration regression from EF Core 8 to 9
I'm in the process of upgrading an application from .NET 8 to 9.
But I'm running into problems when trying to run existing migrations that have been created with EF Core 8 and working for over a year now on multiple versions of EF Core (8.0.4 to 8.0.11)
I have a migration that throws exceptions and can't complete at all on v9.0.0, but if I revert to 8.0.11 it works fine.
It's a series of multiple pretty complex migrations and changes over time, so creating a small repro will take some time, so I hope that's not necessary. Let me know if it is or you need more information!
Include your code
I have Temporal tables enabled for a model: "TestObjective" like this:
At some point we renamed a relations table from "TestObjectivesProjectRoles" to "TestObjectivesOwners".
All this is auto generated with
dotnet-ef migrations add
But the migration generated to alter the temporal columns:
Results in this error:
Include provider and version information
EF Core version: 9.0.0
Database provider: Microsoft.EntityFrameworkCore.SqlServer
Target framework: .NET 9.0
Operating system: Windows & Linux (Local & CICD)
IDE: Visual Studio Enterprise 2022 17.12.0
The text was updated successfully, but these errors were encountered: