[Phase 1][TABLEGROUP][GIN INDEX] GIN Indexes with NO TABELGROUP fails to read data.

[Arjun-yb]

Jira Link: [DB-383](https://yugabyte.atlassian.net/browse/DB-383)
DB Version: 2.11.2 and 2.11.3
Steps:


1. Create tablegroup

2. Create table with tablegroup
3. Create GIN Index with no tablegroup (using NO TABLEGROUP)
4. Load some data
5. Read data from table using index scan

Expected:
 After Step 5, it should return correct result
Actul: After Step 5, it is throwing below ERROR


db_21=# SELECT count(*) FROM employees where introduction @@ to_tsquery('simple', 'the');
ERROR:  Timed out: Read(tablet: 1d0fb617fa234fb8a07c449a1f988577, num_ops: 1, num_attempts: 271, txn: 00000000-0000-0000-0000-000000000000, subtxn: [none]) passed its deadline 1388.135s (passed: 60.044s): Network error (yb/util/net/socket.cc:545): recvmsg got EOF from remote (system error 108)
FATAL:  terminating connection due to administrator command

If user executes any query immediately it looses its connection.

db_21=# select count(*) from employees;
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!>

Found below log in tserver FATAL and ERROR log files

F0121 07:44:02.958986 26754 tablet.cc:3642] Check failed: _s.ok() Bad status: Not found (yb/tablet/tablet_metadata.cc:350): Table <unknown_table_name> (00004000000030008000000000004007) not found in Raft group 1d0fb617fa234fb8a07c449a1f988577

And

F20220121 07:35:03 ../../src/yb/tablet/tablet.cc:3642] Check failed: _s.ok() Bad status: Not found (yb/tablet/tablet_metadata.cc:350): Table <unknown_table_name> (00004000000030008000000000004007) not found in Raft group 1d0fb617fa234fb8a07c449a1f988577
    @     0x7f90ba93335c  yb::LogFatalHandlerSink::send()
    @     0x7f90b9d31fde  google::LogMessage::SendToLog()
    @     0x7f90b9d2f16a  google::LogMessage::Flush()
    @     0x7f90b9d32859  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f90c4a7caa6  yb::tablet::Tablet::GetSchema()
    @     0x7f90c4a59cf2  yb::tablet::AbstractTablet::HandlePgsqlReadRequest()
    @     0x7f90c4a896b3  yb::tablet::Tablet::HandlePgsqlReadRequest()
    @     0x7f90c588d6b6  yb::tserver::TabletServiceImpl::DoReadImpl()
    @     0x7f90c588e7ec  yb::tserver::TabletServiceImpl::DoRead()
    @     0x7f90c588ebac  yb::tserver::TabletServiceImpl::CompleteRead()
    @     0x7f90c5890b4d  yb::tserver::TabletServiceImpl::Read()
    @     0x7f90c1bc8da6  _ZN2yb3rpc10HandleCallINS0_19RpcCallPBParamsImplINS_7tserver13ReadRequestPBENS3_14ReadResponsePBEEEZZNS3_21TabletServerServiceIf11InitMethodsERK13scoped_refptrINS_12MetricEntityEEENKUlSt10shared_ptrINS0_11InboundCallEEE0_clESF_EUlPKS4_PS5_NS0_10RpcContextEE_EEDaSF_T0_
    @     0x7f90c1bc8f5d  _ZNSt17_Function_handlerIFvSt10shared_ptrIN2yb3rpc11InboundCallEEEZNS1_7tserver21TabletServerServiceIf11InitMethodsERK13scoped_refptrINS1_12MetricEntityEEEUlS4_E0_E9_M_invokeERKSt9_Any_dataOS4_
    @     0x7f90c1bbe304  yb::tserver::TabletServerServiceIf::Handle()
    @     0x7f90bb87f69e  yb::rpc::ServicePoolImpl::Handle()
    @     0x7f90bb8219c4  yb::rpc::InboundCall::InboundCallTask::Run()
    @     0x7f90bb8932e8  yb::rpc::(anonymous namespace)::Worker::Execute()
    @     0x7f90ba9d5705  yb::Thread::SuperviseThread()
    @     0x7f90b62b2694  start_thread
    @     0x7f90b59ef41d  __clone
    @              (nil)  (unknown)

Note: Same is working fine with regular indexes.

[jaki]

Here's a small repro:

CREATE TABLEGROUP g;
CREATE TABLE t (i int[]) TABLEGROUP g;
CREATE INDEX ON t USING gin (i) NO TABLEGROUP;
SELECT * FROM t WHERE i @> '{1}';

It's working fine with non-tablegroup colocation and opt-out. It's not possible to opt-out only the index in non-tablegroup colocation. Maybe it has to do with the index being opted out but the table not.

CREATE DATABASE co colocated true;
\c co
CREATE TABLE t (i int[]) WITH (colocated=false);
CREATE INDEX ON t USING gin (i);
SELECT * FROM t WHERE i @> '{1}';

[tverona1]

Will be resolved along with #12503

[tverona1]

We'll restrict indexes to share same topology as base table (per #12503). Moving this to phase 3 in case we want to support this later.

[frozenspider]

Such topology support has been removed in #12503, closing as wontfix

[def-]

@frozenspider @tverona1 Is this expected to be fixed in 2.14? On 2.14.0.0-b70 I am still seeing:

testginindexwithoptouttablegroupandindexbackfill-aws-rf3: Start
    (     0.240s) User Login : Success
    (     0.095s) Refresh YB Version : Success
    (   238.507s) Setup Provider : Success
    (   361.015s) Create universe jenk-is1570-4647d1b09e-20220625-192220 : Success
    (   432.829s) Load data to the table : Success
    (    67.201s) Create GIN Indexes : Success
    (     3.455s) Collect table and indexes' tablets : Success
    (     0.001s) Validate tablets of table and indexes : Success
    (    51.761s) Validate GIN Indexes query plan and data GHI: 11163 : >>> Integration Test Failed <<<
Aborted: Shutdown connection
(    23.661s) Saved server log files and keys at /share/jenkins/workspace/itest-system/logs/2.14.0.0_testginindexwithoptouttablegroupandindexbackfill-aws-rf3_20220625_221408 : Success
(     4.898s) Check for fatals : Hit error before destroying universe 'jenk-is1570-4647d1b09e-20220625-192220':
Found FATAL log file in 10.9.128.38/tserver.
Found FATAL log file in 10.9.201.213/tserver.
Found FATAL log file in 10.9.210.216/tserver.
(    60.169s) Destroy universe : Success
testginindexwithoptouttablegroupandindexbackfill-aws-rf3: End

The FATALs contain:

F20220625 22:13:19 ../../src/yb/tablet/tablet.cc:3814] Check failed: _s.ok() Bad status: Not found (yb/tablet/tablet_metadata.cc:421): Table <unknown_table_name> (0000400500003000800000000000400c) not found in Raft group 2f5e7c1812e14b3eb9ee78b67d426788
    @          0x29fabf9  google::LogDestination::LogToSinks()
    @          0x29fa07b  google::LogMessage::SendToLog()
    @          0x29fb3c3  google::LogMessage::Flush()
    @          0x29fb9df  google::LogMessageFatal::~LogMessageFatal()
    @          0x34f0a77  yb::tablet::Tablet::GetDocReadContext()
    @          0x34b63cb  yb::tablet::AbstractTablet::ProcessPgsqlReadRequest()
    @          0x34f1ba7  yb::tablet::Tablet::HandlePgsqlReadRequest()
    @          0x367da7b  yb::tserver::(anonymous namespace)::ReadQuery::Complete()
    @          0x3678856  yb::tserver::PerformRead()
    @          0x36b5972  yb::tserver::TabletServiceImpl::Read()
    @          0x378c754  std::__1::__function::__func<>::operator()()
    @          0x378dfb6  yb::tserver::TabletServerServiceIf::Handle()
    @          0x344f083  yb::rpc::ServicePoolImpl::Handle()
    @          0x339fe05  yb::rpc::InboundCall::InboundCallTask::Run()
    @          0x345de1d  yb::rpc::(anonymous namespace)::Worker::Execute()
    @          0x3a1c684  yb::Thread::SuperviseThread()
    @     0x7f261650c694  start_thread
    @     0x7f2616a0e41d  __clone

[frozenspider]

@def- no, this restriction has been implemented on master after 2.14 was cut, and since tablegroups are beta feature we didn't consider backporting it.

Also note that while it won't be possible to create index with such weird topology anymore, existing indexes created prior to the restriction would continue to have issues. The solution would be to drop an index and recreate an index (which will be in the same tablegroup).