ddl: Fix generated column can refer only to generated columns defined prior to it. #11549

AilinKid · 2019-07-31T12:26:40Z

What problem does this PR solve?

This PR tries to solve the following bug that caused by the order of generated columns and non-generated columns:

TiDB master:

tidb> create table t (a int, b int as (a + 1), c int as (b + 1))
Query OK, 0 rows affected (0.16 sec)

tidb> insert into t set a=1;
Query OK, 1 rows affected (0.0 sec)

tidb>alter table t add column e int as (c + 1) first;
Query OK, 0 rows affected (0.16 sec)

tidb> select * from t
+--------+---+---+---+
| e      | a | b | c |
+--------+---+---+---+
| <null> | 1 | 2 | 3 |                      
+--------+---+---+---+
1 row in set

tidb> desc select * from t;
+-----------------+----------+------+-------------------------------------------------------------------------------------------------------------------+
| id              | count    | task | operator info                                                                                                     |
+-----------------+----------+------+-------------------------------------------------------------------------------------------------------------------+
| Projection_5    | 10000.00 | root | cast(plus(cast(plus(test.t.b, 1)), 1)), test.t.a, cast(plus(test.t.a, 1)), cast(plus(cast(plus(test.t.a, 1)), 1)) |
| └─TableReader_7 | 10000.00 | root | data:TableScan_6                                                                                                  |
|   └─TableScan_6 | 10000.00 | cop  | table:t, range:[-inf,+inf], keep order:false, stats:pseudo                                                        |
+-----------------+----------+------+-------------------------------------------------------------------------------------------------------------------+

In MySQL 5.7 & 8.0:

mysql> create table t (a int, b int as (a + 1), c int as (b + 1))
Query OK, 0 rows affected (0.16 sec)

mysql> insert into t set a=1;
Query OK, 1 rows affected (0.0 sec)

mysql>alter table t add column e int as (c + 1) first;
(3107, u'Generated column can refer only to generated columns defined prior to it.')

Before putting add generated column to ddl job, computing the generated column offset itself and judging the relation with depended generated column offset as well.

What is changed and how it works?

Find the generated column offset itself in func findPositionRelativeColumn
judge the relation with depended generated column offset in func checkDependedColValid

Check List

Tests

Unit test
Integration test

AilinKid · 2019-07-31T12:46:11Z

/rebuild

AilinKid · 2019-07-31T13:07:53Z

/run-all-tests

codecov · 2019-08-01T07:25:22Z

Codecov Report

Merging #11549 into master will decrease coverage by 0.6576%.
The diff coverage is 87.5%.

@@               Coverage Diff                @@
##             master     #11549        +/-   ##
================================================
- Coverage   81.8655%   81.2079%   -0.6577%     
================================================
  Files           427        426         -1     
  Lines         94759      91874      -2885     
================================================
- Hits          77575      74609      -2966     
- Misses        11830      11895        +65     
- Partials       5354       5370        +16

Deardrops · 2019-08-01T07:48:08Z

There is an implement for detecting if there is a generated column refer to another generated column occurring earlier in the table. Could we multiplex this function?

tidb/ddl/ddl_api.go

Line 836 in e3c9abd

func checkGeneratedColumn(colDefs []*ast.ColumnDef) error {

tidb/ddl/generated_column.go

Lines 38 to 42 in 1c43f55

    
           if attr.generated && attribute.position <= attr.position { 
        
           	// A generated column definition can refer to other 
        
           	// generated columns occurring earlier in the table. 
        
           	err := errGeneratedColumnNonPrior.GenWithStackByArgs() 
        
           	return errors.Trace(err)

tangenta · 2019-08-01T07:34:46Z

ddl/db_integration_test.go

+		rs.Close()
+	}
+	c.Assert(err, NotNil)
+	c.Assert(err.Error(), Equals, "[ddl:1054]Unknown column 'f' in 'generated column function'")


Would you please consider using assertErrorCode()?

Would you please consider using assertErrorCode()?

Addressed @tangenta

ddl/ddl_api.go

tangenta · 2019-08-01T08:26:28Z

executor/ddl_test.go

+	_, err = tk.Exec("alter table t add column(e int as (c+1))")
+	c.Assert(err, IsNil)
+}
+


Why do we need to test the same case twice?

Why do we need to test the same case twice?

Addressed, PTAL

AilinKid · 2019-08-01T13:31:31Z

There is an implement for detecting if there is a generated column refer to another generated column occurring earlier in the table. Could we multiplex this function?

tidb/ddl/ddl_api.go

Line 836 in e3c9abd

func checkGeneratedColumn(colDefs []*ast.ColumnDef) error {

tidb/ddl/generated_column.go

Lines 38 to 42 in 1c43f55

if attr.generated && attribute.position <= attr.position {

// A generated column definition can refer to other

// generated columns occurring earlier in the table.

err := errGeneratedColumnNonPrior.GenWithStackByArgs()

return errors.Trace(err)

Thx very much.
I’ve check it. Logic is same to me, but func is unable to use.
It’s in create table statement, so the depended column parameter is all ast.colDefs.
but here are only columns instead when you in add column statement.

I've changed it: PTAL
To reduce intrusion into original func, I've kept it real.
To be more clear, I've simulated the similar func definition and name.

AilinKid · 2019-08-01T13:35:27Z

There is an implement for detecting if there is a generated column refer to another generated column occurring earlier in the table. Could we multiplex this function?

tidb/ddl/ddl_api.go

Line 836 in e3c9abd

func checkGeneratedColumn(colDefs []*ast.ColumnDef) error {

tidb/ddl/generated_column.go

Lines 38 to 42 in 1c43f55

if attr.generated && attribute.position <= attr.position {

// A generated column definition can refer to other

// generated columns occurring earlier in the table.

err := errGeneratedColumnNonPrior.GenWithStackByArgs()

return errors.Trace(err)

Addressed @Deardrops

ddl/db_integration_test.go

ddl/ddl_api.go

Deardrops · 2019-08-02T05:54:16Z

ddl/generated_column.go

+		_, ok := dependCols[col.Name.L]
+		if col.Offset >= pos && ok && col.IsGenerated() {
+			// Generated column can refer only to generated columns defined prior to it.
+			return errGeneratedColumnNonPrior.GenWithStackByArgs()


if _, ok := dependCols[col.Name.L]; ok { if col.IsGenerated() && col.Offset >= pos { // Generated column can refer only to generated columns defined prior to it. return errGeneratedColumnNonPrior.GenWithStackByArgs() } }

Deardrops · 2019-08-02T05:56:35Z

ddl/generated_column.go

+	if pos == nil {
+		return position, nil
+	}
+	if pos.Tp == ast.ColumnPositionFirst {


suggestion: changing to select ... case ... pattern, for better readability.

It handles like ddl/column.103 when in ddl woker.

Deardrops · 2019-08-02T06:02:11Z

ddl/generated_column.go

@@ -50,10 +50,28 @@ func verifyColumnGeneration(colName2Generation map[string]columnGenerationInDDL,
 	return nil
 }

+// verifyColumnGenerationSingle is for ADD GENERATED COLUMN, we just need verify one column itself.
+func verifyColumnGenerationSingle(dependCols map[string]struct{}, cols []*table.Column, position *ast.ColumnPosition) error {


The first parameter of this function is dependCols, but when call this function, you passed the parameter named duplicateColNames, is them same?
https://github.com/pingcap/tidb/blob/a7cf2762510cb1f4efd14d6ddd1ae08558ded4e8/ddl/ddl_api.go#L2113

Same. Since dependCols will change in checkDependedColExist, so I make it copy.
Parameter Addressed.

AilinKid · 2019-08-02T06:40:42Z

/run-all-tests

Deardrops · 2019-08-02T06:49:59Z

LGTM

tangenta

LGTM

ngaut · 2019-08-03T08:57:32Z

ddl/ddl_api.go

@@ -2101,8 +2100,17 @@ func (d *ddl) AddColumn(ctx sessionctx.Context, ti ast.Ident, spec *ast.AlterTab
 			if err = checkAutoIncrementRef(specNewColumn.Name.Name.L, dependColNames, t.Meta()); err != nil {
 				return errors.Trace(err)
 			}
+			duplicateColNames := make(map[string]struct{})


Make map with capacity.

ddl/ddl_api.go

ddl/db_integration_test.go

ddl/generated_column.go

bb7133 · 2019-08-06T07:01:01Z

ddl/generated_column.go

+// findPositionRelativeColumn return a pos relative to added generated column position
+func findPositionRelativeColumn(cols []*table.Column, pos *ast.ColumnPosition) (int, error) {
+	position := len(cols)
+	// get column position default is append behind


Suggested change

// get column position default is append behind

// get column position default is append behind

What does this comment mean? I'm confused

Addressed.
It's should be: Get column position, the default is cols's length means appending behind.

bb7133 · 2019-08-06T07:02:29Z

ddl/generated_column.go

+		if col == nil {
+			return -1, ErrBadField.GenWithStackByArgs(pos.RelativeColumn, "generated column function")
+		}
+		// Insert position is after the mentioned column.


position cannot be after a column, you can say 'Inserted column is after the mentioned column'.

Addressed. Yes, shouldn't be a verb here.

ddl/generated_column.go

bb7133 · 2019-08-08T03:49:55Z

/run-all-tests

sre-bot · 2019-08-08T04:00:55Z

cherry pick to release-3.0 failed

sre-bot · 2019-08-08T04:02:36Z

cherry pick to release-2.1 failed

…d prior to it. (#11549) (#11686)

AilinKid changed the title ~~Fix generated column can refer only to generated columns defined prior to it.~~ ddl: Fix generated column can refer only to generated columns defined prior to it. Jul 31, 2019

AilinKid added the type/bugfix This PR fixes a bug. label Jul 31, 2019

AilinKid force-pushed the fix_bug_generated_column branch from d11b415 to cdfd042 Compare August 1, 2019 07:20

tangenta reviewed Aug 1, 2019

View reviewed changes

AilinKid force-pushed the fix_bug_generated_column branch from bd67d0a to 6a89724 Compare August 1, 2019 13:32

tangenta reviewed Aug 2, 2019

View reviewed changes

ddl/db_integration_test.go Outdated Show resolved Hide resolved

ddl/ddl_api.go Outdated Show resolved Hide resolved

Deardrops reviewed Aug 2, 2019

View reviewed changes

AilinKid force-pushed the fix_bug_generated_column branch from 1700abd to 87dfdb3 Compare August 2, 2019 06:43

AilinKid added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 2, 2019

tangenta reviewed Aug 2, 2019

View reviewed changes

AilinKid added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 2, 2019

ngaut requested a review from zimulala August 3, 2019 08:56

ngaut reviewed Aug 3, 2019

View reviewed changes

AilinKid added the status/all tests passed label Aug 6, 2019

winkyao added the component/DDL-need-LGT3 label Aug 6, 2019

AilinKid added needs-cherry-pick-2.1 and removed component/DDL-need-LGT3 labels Aug 6, 2019

bb7133 reviewed Aug 6, 2019

View reviewed changes

AilinKid force-pushed the fix_bug_generated_column branch from 6f1fbc8 to f5f1817 Compare August 6, 2019 07:42

AilinKid added 16 commits August 8, 2019 11:29

modify comments

f3e0faf

add unit test

9c72841

fix make dev panic

cef78e2

add union test in case of code coverage decrease

f6bd2d0

be more clear in func

cf5953e

use assertErrorCode to test

20f79eb

delete duplicate test case

8048985

address extract t.cols() out avoiding func calls

2ed5cc4

code style be more clear

d027b0d

parameter be clear to understand

cc5204d

Make map with capacity

390f99e

modify comment

6aa8def

modify comments

596a859

modify two comment

52b617d

modify a comment

c2d0fe7

delete redundant comment

396ef77

AilinKid dismissed bb7133’s stale review via 396ef77 August 8, 2019 03:44

AilinKid force-pushed the fix_bug_generated_column branch from c99a167 to 396ef77 Compare August 8, 2019 03:44

AilinKid added status/LGT3 The PR has already had 3 LGTM. and removed status/LGT2 Indicates that a PR has LGTM 2. labels Aug 8, 2019

bb7133 approved these changes Aug 8, 2019

View reviewed changes

bb7133 merged commit 110b073 into pingcap:master Aug 8, 2019

This was referenced Aug 8, 2019

ddl : Fix generated column can refer only to generated columns defined prior to it. (#11549) #11682

Merged

ddl : Fix generated column can refer only to generated columns defined prior to it. (#11549) #11686

Merged

sre-bot pushed a commit that referenced this pull request Aug 12, 2019

ddl : Fix generated column can refer only to generated columns define…

21c5e45

…d prior to it. (#11549) (#11686)

you06 added the sig/sql-infra SIG: SQL Infra label Mar 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: Fix generated column can refer only to generated columns defined prior to it. #11549

ddl: Fix generated column can refer only to generated columns defined prior to it. #11549

AilinKid commented Jul 31, 2019 •

edited by bb7133

Loading

AilinKid commented Jul 31, 2019

AilinKid commented Jul 31, 2019

codecov bot commented Aug 1, 2019 •

edited

Loading

Deardrops commented Aug 1, 2019 •

edited

Loading

tangenta Aug 1, 2019

AilinKid Aug 1, 2019

tangenta Aug 1, 2019

AilinKid Aug 1, 2019

AilinKid commented Aug 1, 2019 •

edited

Loading

AilinKid commented Aug 1, 2019 •

edited

Loading

Deardrops Aug 2, 2019

AilinKid Aug 2, 2019

Deardrops Aug 2, 2019

AilinKid Aug 2, 2019

Deardrops Aug 2, 2019

AilinKid Aug 2, 2019 •

edited

Loading

AilinKid commented Aug 2, 2019

Deardrops commented Aug 2, 2019

tangenta left a comment

ngaut Aug 3, 2019

AilinKid Aug 5, 2019

bb7133 Aug 6, 2019

AilinKid Aug 6, 2019

bb7133 Aug 6, 2019

AilinKid Aug 6, 2019 •

edited

Loading

bb7133 commented Aug 8, 2019

sre-bot commented Aug 8, 2019

sre-bot commented Aug 8, 2019

	// get column position default is append behind
	// get column position default is append behind

ddl: Fix generated column can refer only to generated columns defined prior to it. #11549

ddl: Fix generated column can refer only to generated columns defined prior to it. #11549

Conversation

AilinKid commented Jul 31, 2019 • edited by bb7133 Loading

What problem does this PR solve?

What is changed and how it works?

Check List

AilinKid commented Jul 31, 2019

AilinKid commented Jul 31, 2019

codecov bot commented Aug 1, 2019 • edited Loading

Codecov Report

Deardrops commented Aug 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid commented Aug 1, 2019 • edited Loading

AilinKid commented Aug 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid Aug 2, 2019 • edited Loading

Choose a reason for hiding this comment

AilinKid commented Aug 2, 2019

Deardrops commented Aug 2, 2019

tangenta left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AilinKid Aug 6, 2019 • edited Loading

Choose a reason for hiding this comment

bb7133 commented Aug 8, 2019

sre-bot commented Aug 8, 2019

sre-bot commented Aug 8, 2019

AilinKid commented Jul 31, 2019 •

edited by bb7133

Loading

codecov bot commented Aug 1, 2019 •

edited

Loading

Deardrops commented Aug 1, 2019 •

edited

Loading

AilinKid commented Aug 1, 2019 •

edited

Loading

AilinKid commented Aug 1, 2019 •

edited

Loading

AilinKid Aug 2, 2019 •

edited

Loading

AilinKid Aug 6, 2019 •

edited

Loading