Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PY-client] Unify setup.py and requirement.txt #12601

Merged
merged 4 commits into from
Jun 6, 2024
Merged

[PY-client] Unify setup.py and requirement.txt #12601

merged 4 commits into from
Jun 6, 2024

Conversation

HTHou
Copy link
Contributor

@HTHou HTHou commented May 28, 2024

Description

#12591

@codecov Codecov
Copy link

codecov bot commented May 28, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 42.15%. Comparing base (baf37c3) to head (22ef997).
Report is 65 commits behind head on master.

Additional details and impacted files 
@@             Coverage Diff              @@
##             master   #12601      +/-   ##
============================================
- Coverage     43.12%   42.15%   -0.97%     
- Complexity       69       71       +2     
============================================
  Files          3188     3258      +70     
  Lines        198006   205310    +7304     
  Branches      23557    24477     +920     
============================================
+ Hits          85392    86552    +1160     
- Misses       112614   118758    +6144

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@HTHou HTHou mentioned this pull request Jun 6, 2024
Closed
2 tasks
@ycycse
Copy link
Member

ycycse commented Jun 6, 2024

I find the version requirement in requirement.txt is
# Pandas Export pandas>=1.3.5
Should we unify these?

@ycycse
Copy link
Member

ycycse commented Jun 6, 2024 •
edited
Loading

I find the version requirement in requirement.txt is # Pandas Export pandas>=1.3.5 Should we unify these?

By the way, other requirements in requirement.txt are

pandas>=1.3.5
numpy>=1.21.4
thrift>=0.14.1

sqlalchemy == 1.3.20
sqlalchemy-utils == 0.36.8

which are not the same as the requirement in setup.py.

Generally speaking, the version in setup.py determines the specific version of dependencies that users will download when installing our package. The requirements.txt is typically the version that developers pay attention to. Although inconsistency will not affect usage, in order to unify the development and application environment, I think it is better to unify the two (or let requirements.txt include what setup specifies).

@HTHou
Copy link
Contributor Author

HTHou commented Jun 6, 2024

I find the version requirement in requirement.txt is # Pandas Export pandas>=1.3.5 Should we unify these?

Yes, it makes sense. Fixed.

@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Jun 6, 2024

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

@HTHou HTHou changed the title Remove the restrict of pandas < 2 Unify setup.py and requirement.txt Jun 6, 2024
@HTHou HTHou changed the title Unify setup.py and requirement.txt [PY-client] Unify setup.py and requirement.txt Jun 6, 2024
ycycse
ycycse approved these changes Jun 6, 2024
View reviewed changes
Copy link
Member

@ycycse ycycse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM~

@HTHou HTHou merged commit 849c09f into master Jun 6, 2024
37 of 40 checks passed
@HTHou HTHou deleted the HTHou-patch-2 branch June 6, 2024 09:36
@rmpinchback
Copy link

I don't have any opinions on the sqlalchemy version range because I haven't had reason know the v2 changes or how that could relate to the iotdb python client, but as for the pandas and numpy changes, that all looks good to me. Thanks!

chrisdutz pushed a commit to chrisdutz/iotdb that referenced this pull request Jun 7, 2024
@HTHou @chrisdutz
[PY-client] Unify setup.py and requirement.txt (apache#12601)
47ea7f8
JackieTien97 added a commit that referenced this pull request Jun 9, 2024
@JackieTien97 @Cpaulyz
@VGalaxies @liyuheng55555 @shuwenwei @HxpSerein @SteveYurongSu @ycycse @HTHou @Caideyipi @clayburn @2b3c511 @DanielWang2035 @lancelly @Sh-Zh-7
Merge master0607 (#12689)
8c5ea36 
* [IOTDB-6336] Add max retry time duration and whether to retry for unknown errors configurations

* Optimize error message when creating unsupported data type (#12650)

Optimize error message when creating unsupported data type

* Pipe: fix uninitialized config when validate PipeHistoricalDataRegionTsFileExtractor parameters & Subscription: avoid poll messages from unsubscribed topics (#12648)

* Pipe: fix uninitialized config when validate PipeHistoricalDataRegionTsFileExtractor parameters

* Subscription: avoid poll messages from unsubscribed topics

* Fix region migration's timeout judgement #12639

* fix compaction recover skip negative time partition (#12637)

* Fix Concurrency Instability When IoTConsensus LogDispatcher Exits #12655

* JDBC: Fix IndexOutOfBoundsException when calling getBytes (#12662)

* JDBC: Fix ArrayOutOfBoundException when calling getBytes

* Update JDBCCharsetExample.java

* [IOTDB-6335] Redundant rows when using GROUP BY TIME with LIMIT

* Pipe: Enhance error handling logic in pipe async connector's handlers (#12669)

* Print log when read resource error (#12666)

* Load: Make load SQL option VERIFY can only be set to true (#12670)

* Pipe: Improved hybrid algorithm and avoid the caculations of tsfiles in connector pending queue tsfiles to block the forwarding of realtime requests (#12668)

* Migrate to Develocity Maven Extension (#12658)

* Migrate to Develocity Maven Extension
* Update to Common Custom User Data Maven Extension 2.0

* Update import schema logic implementation and optimize -help display information (#12643)

* add import-schema.sh/bat and export-schema.sh/bat in cli/tools

* fix import schema

* fix import schema

---------

Co-authored-by: 2b3c511 <rong.li@timecho.com>

* Subscription: fix topic now timestamp precision (#12663)

* [IOTDB-6337] Refine the count calculation in RegionScan framework

* Bump version of master branch to 1.3.3-SNAPSHOT (#12672)

* Pipe: Reduce warn log when PipeConfigRegionExtractorMetrics does not exist (#12673)

* [IOTDB-6061] Fix the instability failure caused by initServer in IoTConsensus UT not binding to the corresponding port (#12674)

* fix concurrent

* fix concurrent

* Load: Make load SQL option VERIFY can only be set to true (A follow-up fix for #12670) (#12676)

* Pipe: add compression level config for connector ZSTD compressor (#12630)

Co-authored-by: Steve Yurong Su <rong@apache.org>

* [PY-client] Unify setup.py and requirement.txt (#12601)

* Pipe IT: Ignore failed cases caused by cluster restart failure (#12678)

* update import and export data -help description (#12677)

Co-authored-by: 2b3c511 <rong.li@timecho.com>

* Pipe: Fix potential NPE when lastEvent == null in PipeConnectorSubtask (#12680)

* Fix wrong calculation for maxBytesCanReserve for IdentitySinkOperator

* Change floating point encoding method in IT

* Throw `IndexOutOfBound` exception in row implementation.

* change version to 1.3.3-SNAPSHOT for iotdb-relational-grammar

* change version to 1.3.3-SNAPSHOT for datanode dependency on iotdb-relational-grammar

---------

Co-authored-by: Chen YZ <43774645+Cpaulyz@users.noreply.github.com>
Co-authored-by: V_Galaxy <vgalaxies@apache.org>
Co-authored-by: Li Yu Heng <liyuheng55555@126.com>
Co-authored-by: shuwenwei <55970239+shuwenwei@users.noreply.github.com>
Co-authored-by: Xiangpeng Hu <65238551+BUAAserein@users.noreply.github.com>
Co-authored-by: Steve Yurong Su <rong@apache.org>
Co-authored-by: YangCaiyin <ycycse@gmail.com>
Co-authored-by: Haonan <hhaonan@outlook.com>
Co-authored-by: Caideyipi <87789683+Caideyipi@users.noreply.github.com>
Co-authored-by: Clay Johnson <cjohnson@gradle.com>
Co-authored-by: Summer <43237967+2b3c511@users.noreply.github.com>
Co-authored-by: 2b3c511 <rong.li@timecho.com>
Co-authored-by: Zikun Ma <55695098+DanielWang2035@users.noreply.github.com>
Co-authored-by: Liao Lanyu <1435078631@qq.com>
Co-authored-by: Zhihao Shen <zhihaoshen7@qq.com>
JackieTien97 added a commit that referenced this pull request Jun 11, 2024
@JackieTien97 @Cpaulyz
@VGalaxies @liyuheng55555 @shuwenwei @HxpSerein @SteveYurongSu @ycycse @HTHou @Caideyipi @clayburn @2b3c511 @DanielWang2035 @lancelly @Sh-Zh-7 @CRZbulabula @OneSizeFitsQuorum @ppppoooo @CritasWang @Beyyes
Merge master0611 (#12703)
d72429f 
* [IOTDB-6336] Add max retry time duration and whether to retry for unknown errors configurations

* Optimize error message when creating unsupported data type (#12650)

Optimize error message when creating unsupported data type

* Pipe: fix uninitialized config when validate PipeHistoricalDataRegionTsFileExtractor parameters & Subscription: avoid poll messages from unsubscribed topics (#12648)

* Pipe: fix uninitialized config when validate PipeHistoricalDataRegionTsFileExtractor parameters

* Subscription: avoid poll messages from unsubscribed topics

* Fix region migration's timeout judgement #12639

* fix compaction recover skip negative time partition (#12637)

* Fix Concurrency Instability When IoTConsensus LogDispatcher Exits #12655

* JDBC: Fix IndexOutOfBoundsException when calling getBytes (#12662)

* JDBC: Fix ArrayOutOfBoundException when calling getBytes

* Update JDBCCharsetExample.java

* [IOTDB-6335] Redundant rows when using GROUP BY TIME with LIMIT

* Pipe: Enhance error handling logic in pipe async connector's handlers (#12669)

* Print log when read resource error (#12666)

* Load: Make load SQL option VERIFY can only be set to true (#12670)

* Pipe: Improved hybrid algorithm and avoid the caculations of tsfiles in connector pending queue tsfiles to block the forwarding of realtime requests (#12668)

* Migrate to Develocity Maven Extension (#12658)

* Migrate to Develocity Maven Extension
* Update to Common Custom User Data Maven Extension 2.0

* Update import schema logic implementation and optimize -help display information (#12643)

* add import-schema.sh/bat and export-schema.sh/bat in cli/tools

* fix import schema

* fix import schema

---------

Co-authored-by: 2b3c511 <rong.li@timecho.com>

* Subscription: fix topic now timestamp precision (#12663)

* [IOTDB-6337] Refine the count calculation in RegionScan framework

* Bump version of master branch to 1.3.3-SNAPSHOT (#12672)

* Pipe: Reduce warn log when PipeConfigRegionExtractorMetrics does not exist (#12673)

* [IOTDB-6061] Fix the instability failure caused by initServer in IoTConsensus UT not binding to the corresponding port (#12674)

* fix concurrent

* fix concurrent

* Load: Make load SQL option VERIFY can only be set to true (A follow-up fix for #12670) (#12676)

* Pipe: add compression level config for connector ZSTD compressor (#12630)

Co-authored-by: Steve Yurong Su <rong@apache.org>

* [PY-client] Unify setup.py and requirement.txt (#12601)

* Pipe IT: Ignore failed cases caused by cluster restart failure (#12678)

* update import and export data -help description (#12677)

Co-authored-by: 2b3c511 <rong.li@timecho.com>

* Pipe: Fix potential NPE when lastEvent == null in PipeConnectorSubtask (#12680)

* Fix wrong calculation for maxBytesCanReserve for IdentitySinkOperator

* Change floating point encoding method in IT

* Throw `IndexOutOfBound` exception in row implementation.

* Thread safely SeriesPartitionTable (#12679)

* finish

* use concurrentskiplist

* bug fix

---------

Co-authored-by: OneSizeFitQuorum <tanxinyu@apache.org>

* Pipe: Refactor some features for user access (#12686)

* Set timestamp precision same as time column (#12681)

* done

* done

* rollback change

* fix IT

* fix IT

* Set configuration on node (#12626)

Co-authored-by: Haonan <hhaonan@outlook.com>

* Fix cli report error when ignoreTimestamp is true (#12691)

* done

* done

* rollback change

* fix IT

* fix IT

* done

* invoke ci

* done

* invoke ci

* Load: Add check for BufferUnderflowException (#12690)

Co-authored-by: xz m <xiangzhi.meng@timecho.com>

* remove excess stack

* Fix new UDTF execution framework dead loop error.

* Perfect methods of IAnalysis to adapt the write process of table model

* add WriteFragmentParallelPlanner in TableModelDistributePlanner

---------

Co-authored-by: Chen YZ <43774645+Cpaulyz@users.noreply.github.com>
Co-authored-by: V_Galaxy <vgalaxies@apache.org>
Co-authored-by: Li Yu Heng <liyuheng55555@126.com>
Co-authored-by: shuwenwei <55970239+shuwenwei@users.noreply.github.com>
Co-authored-by: Xiangpeng Hu <65238551+BUAAserein@users.noreply.github.com>
Co-authored-by: Steve Yurong Su <rong@apache.org>
Co-authored-by: YangCaiyin <ycycse@gmail.com>
Co-authored-by: Haonan <hhaonan@outlook.com>
Co-authored-by: Caideyipi <87789683+Caideyipi@users.noreply.github.com>
Co-authored-by: Clay Johnson <cjohnson@gradle.com>
Co-authored-by: Summer <43237967+2b3c511@users.noreply.github.com>
Co-authored-by: 2b3c511 <rong.li@timecho.com>
Co-authored-by: Zikun Ma <55695098+DanielWang2035@users.noreply.github.com>
Co-authored-by: Liao Lanyu <1435078631@qq.com>
Co-authored-by: Zhihao Shen <zhihaoshen7@qq.com>
Co-authored-by: Yongzao <yongzao@apache.org>
Co-authored-by: OneSizeFitQuorum <tanxinyu@apache.org>
Co-authored-by: ppppoooo <1214343862@qq.com>
Co-authored-by: xz m <xiangzhi.meng@timecho.com>
Co-authored-by: CritasWang <critas@outlook.com>
Co-authored-by: Beyyes <cgf1993@foxmail.com>
JackieTien97 added a commit that referenced this pull request Jun 21, 2024
@JackieTien97 @Cpaulyz
@VGalaxies @liyuheng55555 @shuwenwei @HxpSerein @SteveYurongSu @ycycse @HTHou @Caideyipi @clayburn @2b3c511 @DanielWang2035 @lancelly @Sh-Zh-7 @CRZbulabula @OneSizeFitsQuorum @ppppoooo @CritasWang @Beyyes @MiniSho @SpriCoder @chrisdutz @liuminghui233 @jt2594838 @yschengzi @Pengzna @choubenson @THUMarkLau @SzyWilliam
Mergemaster0620 (#12779)
0343fc2 
* [IOTDB-6336] Add max retry time duration and whether to retry for unknown errors configurations

* Optimize error message when creating unsupported data type (#12650)

Optimize error message when creating unsupported data type

* Pipe: fix uninitialized config when validate PipeHistoricalDataRegionTsFileExtractor parameters & Subscription: avoid poll messages from unsubscribed topics (#12648)

* Pipe: fix uninitialized config when validate PipeHistoricalDataRegionTsFileExtractor parameters

* Subscription: avoid poll messages from unsubscribed topics

* Fix region migration's timeout judgement #12639

* fix compaction recover skip negative time partition (#12637)

* Fix Concurrency Instability When IoTConsensus LogDispatcher Exits #12655

* JDBC: Fix IndexOutOfBoundsException when calling getBytes (#12662)

* JDBC: Fix ArrayOutOfBoundException when calling getBytes

* Update JDBCCharsetExample.java

* [IOTDB-6335] Redundant rows when using GROUP BY TIME with LIMIT

* Pipe: Enhance error handling logic in pipe async connector's handlers (#12669)

* Print log when read resource error (#12666)

* Load: Make load SQL option VERIFY can only be set to true (#12670)

* Pipe: Improved hybrid algorithm and avoid the caculations of tsfiles in connector pending queue tsfiles to block the forwarding of realtime requests (#12668)

* Migrate to Develocity Maven Extension (#12658)

* Migrate to Develocity Maven Extension
* Update to Common Custom User Data Maven Extension 2.0

* Update import schema logic implementation and optimize -help display information (#12643)

* add import-schema.sh/bat and export-schema.sh/bat in cli/tools

* fix import schema

* fix import schema

---------

Co-authored-by: 2b3c511 <rong.li@timecho.com>

* Subscription: fix topic now timestamp precision (#12663)

* [IOTDB-6337] Refine the count calculation in RegionScan framework

* Bump version of master branch to 1.3.3-SNAPSHOT (#12672)

* Pipe: Reduce warn log when PipeConfigRegionExtractorMetrics does not exist (#12673)

* [IOTDB-6061] Fix the instability failure caused by initServer in IoTConsensus UT not binding to the corresponding port (#12674)

* fix concurrent

* fix concurrent

* Load: Make load SQL option VERIFY can only be set to true (A follow-up fix for #12670) (#12676)

* Pipe: add compression level config for connector ZSTD compressor (#12630)

Co-authored-by: Steve Yurong Su <rong@apache.org>

* [PY-client] Unify setup.py and requirement.txt (#12601)

* Pipe IT: Ignore failed cases caused by cluster restart failure (#12678)

* update import and export data -help description (#12677)

Co-authored-by: 2b3c511 <rong.li@timecho.com>

* Pipe: Fix potential NPE when lastEvent == null in PipeConnectorSubtask (#12680)

* Fix wrong calculation for maxBytesCanReserve for IdentitySinkOperator

* Change floating point encoding method in IT

* Throw `IndexOutOfBound` exception in row implementation.

* Thread safely SeriesPartitionTable (#12679)

* finish

* use concurrentskiplist

* bug fix

---------

Co-authored-by: OneSizeFitQuorum <tanxinyu@apache.org>

* Pipe: Refactor some features for user access (#12686)

* Set timestamp precision same as time column (#12681)

* done

* done

* rollback change

* fix IT

* fix IT

* Set configuration on node (#12626)

Co-authored-by: Haonan <hhaonan@outlook.com>

* Fix cli report error when ignoreTimestamp is true (#12691)

* done

* done

* rollback change

* fix IT

* fix IT

* done

* invoke ci

* done

* invoke ci

* Load: Add check for BufferUnderflowException (#12690)

Co-authored-by: xz m <xiangzhi.meng@timecho.com>

* remove excess stack

* Fix new UDTF execution framework dead loop error.

* Perfect methods of IAnalysis to adapt the write process of table model

* Load: Support storing piecenode in multiple folders during 2nd phase (#12675)

Co-authored-by: Steve Yurong Su <rong@apache.org>

* Pipe: add decompressed length in RPC compression payload to avoid potential OOM on receiver (#12701)

* Load: Update loadTsFileDirs after all newLoadTsFileDirs are generated to avoid undefined behavior (#12712)

* Pipe: Fixed NPE in schema pattern parsing & Improved connector subtask retry logic and logs & Fixed the executor used by consensus pipe (#12704)

* Procedure: Fix ProcedureInfo on CN leader may decrease its procId (#12711)

* Fix missing of linux memory metrics (#12713)

* fix missing memory metric

* remove the print of type

* update import and export tsfile description (#12684)

Co-authored-by: 2b3c511 <rong.li@timecho.com>

* Subscription: skip on setup and cluster failure when running subscription restart IT & fix some bugs in SubscriptionExecutorServiceManager (#12710)

* Update pom.xml to fix security issue (#12705)

Co-authored-by: Christofer Dutz <cdutz@apache.org>

* Rename IoTV2Consensus to IoTConsensusV2 (#12715)

Signed-off-by: OneSizeFitQuorum <tanxinyu@apache.org>

* Fix NumberFormatException when upgrade from old version #12719

* Replenish effective mode in iotdb-system.properties (#12706)

* load-tsfile script: disable redirection & load: handle exceptions using the Analysis objects instead of throwing exceptions & pipe: handle SYSTEM_READ_ONLY correctly on receiver side (#12716)

* Subscription: improve parsing logic when using JAVA SDK client & refactor subscription IT & intro `getSubscribedTopicNames` API (#12721)

* Pipe: Fix schema events can not report & Fix delete data events in data regions may fail to mark at schema metrics (#12722)

* Pipe: Support `"source.history.loose-range" = "path"` in iotdb-source  (#12651)

Co-authored-by: Steve Yurong Su <rong@apache.org>

* [IOTDB-6338] Fix wrong query result while using some value filter with LIMIT/OFFSET

* Pipe: fix threads of IoTDB-Pipe-Processor-Executor-Pool stucked by PipeTsFileInsertionEvent#waitForTsFileClose (#12727)

* [ISSUE 12499] Reject query with massive time span in limited memroy

* Fix/include thrift lib and headers in cpp client (#12734)

* fix: Hopefully fixed the build of the cpp-example on windows systems

* fix: Increase the timeout even more to finally make the build succeed.

* fix: Fix the build.

* chore: Fixed the assembly descriptor of the cpp-client to include the thrift libs and header files.

* Pipe / Load / Subscription: Support new TsDataTypes (STRING / BLOB / TIMESTAMP / DATE) (#12665)

* Optimize the log printing when multiple error occur (#12732)

* Update packege hierachy in Client-cpp readme (#12736)

* Support agg(*) query in templated align by device situation

* fix config node use IoTDBDescriptor (#12730)

* PipeConsensus: Delete WAL after dropping database & fix receiver directory recovery (#12738)

* fix wal deletion & create receiver dir

* update consensus protocol class

* merge judge

* modify directory path

* modify directory path

* Add instructions of compiling client-cpp with VS2019. (#12739)

* Subscription: adapt pipe completion signal for automatically drop subscription (#12724)

* Verify connection && Refactor async connection framework (#12667)

* PipeConsensus: add metrics and fix some bugs for pipeConsensus (#12723)

* Decrease TTL Deletion in compaction modification cache (#12687)

* decrease TTL Deletion in compaction modification cache

* Update iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/MultiTsFileDeviceIterator.java

Co-authored-by: Jiang Tian <jt2594838@163.com>

* modify MultiTsFileDeviceIterator

* fix spotless

---------

Co-authored-by: Jiang Tian <jt2594838@163.com>

* Stabilize verify connection IT  (#12745)

* done

* more !

* Fix compaction npe when set modified status (#12729)

* fix compaction npe when set modified status

* check empty page

* Subscription IT: execute jstack for potential stuck test cases (#12728)

* Pipe: Smoothed the rate in pipe's remaining time calculations (#12699)

* Pipe: Fix PipeDataRegionEventCounter not work in PipeRealtimeDataRegionExtractor (#12747)

* Fix python CI (#12754)

* Support show device ttl on path pattern (#12709)

* support show ttl on pathPattern

* fix UT

* modify details

* Support WAL Compression (#12476)

* enable wal compression

remove metrics in mem table flush task, cache hash code in partial path, use gzip to compress wal

batch update metrics

* fix bug

* fix compilation problem

* remove useless code

* recover some code

* support compression type in WAL Compress Header

* support multi version WAL

* edit configuration item

* add log for WAL size

* temp for debug

* fix bug

* remove useless log

* remove one configuration

* use compression rate to update wal disk usage

* fix ut

* fix test

* set default to uncompress

* fix wal ut

* optimize calculating of wal size

* close wal file when the origin size of wal buffer is larger than threshold

* add the size of magic string

* may be fix the bug

* fix with comment

* edit with review

* fix test

* add test for wal compression

* add hot reload

* clean the code to make it more readable

* reuse the byte buffer if possible

* Indicate the encoding of String

* Edit according to comment

* spotless

* Optimize the devicePath generation efficiency of PlanNode when deserializing (#12749)

* Pipe: Support "source.realtime.loose-range" = "path" in iotdb-source (#12751)

* Support active schema query (#12718)

* Check duplicate in start-node and stop-node scripts (#12756)

* check duplicate key in start node script

* modify stop node scripts

* use \n as line separator

* fix config node send showConfiguration to config node classCastException

* [IOTDB-6339] Optimize the time slice control of SeriesScanOperator and AlignedSeriesScanOperator

* Optimize node startup process & Normalize system.properties management (#12752)

* [IOTDB-6340] Clear Coordinator.queryExecutionMap while encountering MemoryNotEnough Exception

* [IOTDB-6342] Clear FragmentInstanceManager.dataNodeQueryContextMap while encountering MemoryNotEnough Exception (#12762)

* Refactor node registration check (#12766)

* done

* done

* Fix overlap between tsfile is not correctly marked (#12748)

* file fileOverlap is not correctly marked

* fix repair compaction may generate too large chunk

* Subscription: support topic loose range for path and time  (#12760)

* Add clean logic for FragmentInstance in case that callback is not added. (#12768)

* SystemPropertiesHandler use Files::move instead of File::renameTo (#12770)

* Files.move

* ?

* ?

* ?

* Pipe: Filter devices by pattern before reading device metadata from TsFile (#12765)

Currently we read the metadata of all devices and measurements when constructing a TsFileInsertionDataContainer. This is both time-consuming and memory-wasting if only a few devices match the pattern. This PR filters the devices by pattern before reading the metadata of devices and measurements, saving memory and I/O cost when there are many unmatched devices.

Note: this only works when TsFile metadata are not cached (due to high memory usage of TsFile metadata cache), because cached metadata can not be filtered so that pipes with arbitrary patterns can use it.

---------

Co-authored-by: Steve Yurong Su <rong@apache.org>

* Pipe: Adjusted some loggers of metrics to avoid unnecessary warns & Include "lastEvent" into pipe's event count metrics & Removed the "userConflict" judgment to data sync failure caused by METADATA_ERROR (#12758)

* Synchronize takeSnapshotAsync on RaftGroupID (#12767)

* SystemPropertiesHandler close reader and writer #12772

* Pipe Consensus: Fix events in connector are not in order because of parameter `'realtime-first' = 'true'` (#12773)

* Pipe: Avoid event being continuously rate-limited even its pipe is dropped (#12753)

Co-authored-by: Steve Yurong Su <rong@apache.org>

* Load: Add some load metrics of time cost, write point and disk throughput (#12735)

Co-authored-by: Steve Yurong Su <rong@apache.org>

---------

Signed-off-by: OneSizeFitQuorum <tanxinyu@apache.org>
Co-authored-by: Chen YZ <43774645+Cpaulyz@users.noreply.github.com>
Co-authored-by: V_Galaxy <vgalaxies@apache.org>
Co-authored-by: Li Yu Heng <liyuheng55555@126.com>
Co-authored-by: shuwenwei <55970239+shuwenwei@users.noreply.github.com>
Co-authored-by: Xiangpeng Hu <65238551+BUAAserein@users.noreply.github.com>
Co-authored-by: Steve Yurong Su <rong@apache.org>
Co-authored-by: YangCaiyin <ycycse@gmail.com>
Co-authored-by: Haonan <hhaonan@outlook.com>
Co-authored-by: Caideyipi <87789683+Caideyipi@users.noreply.github.com>
Co-authored-by: Clay Johnson <cjohnson@gradle.com>
Co-authored-by: Summer <43237967+2b3c511@users.noreply.github.com>
Co-authored-by: 2b3c511 <rong.li@timecho.com>
Co-authored-by: Zikun Ma <55695098+DanielWang2035@users.noreply.github.com>
Co-authored-by: Liao Lanyu <1435078631@qq.com>
Co-authored-by: Zhihao Shen <zhihaoshen7@qq.com>
Co-authored-by: Yongzao <yongzao@apache.org>
Co-authored-by: OneSizeFitQuorum <tanxinyu@apache.org>
Co-authored-by: ppppoooo <1214343862@qq.com>
Co-authored-by: xz m <xiangzhi.meng@timecho.com>
Co-authored-by: CritasWang <critas@outlook.com>
Co-authored-by: Beyyes <cgf1993@foxmail.com>
Co-authored-by: Itami Sho <42286868+MiniSho@users.noreply.github.com>
Co-authored-by: ZhangHongYin <46039728+SpriCoder@users.noreply.github.com>
Co-authored-by: Christofer Dutz <cdutz@apache.org>
Co-authored-by: liuminghui233 <36565497+liuminghui233@users.noreply.github.com>
Co-authored-by: Jiang Tian <jt2594838@163.com>
Co-authored-by: yschengzi <87161145+yschengzi@users.noreply.github.com>
Co-authored-by: Peng Junzhi <78788603+Pengzna@users.noreply.github.com>
Co-authored-by: 周沛辰 <45144903+choubenson@users.noreply.github.com>
Co-authored-by: Liu Xuxin <37140360+THUMarkLau@users.noreply.github.com>
Co-authored-by: William Song <48054931+SzyWilliam@users.noreply.github.com>
HTHou added a commit that referenced this pull request Jun 27, 2024
@HTHou
[PY-client] Unify setup.py and requirement.txt (#12601)
b9ed80b
SzyWilliam pushed a commit to SzyWilliam/iotdb that referenced this pull request Nov 26, 2024
@HTHou @SzyWilliam
[PY-client] Unify setup.py and requirement.txt (apache#12601)
c678daa
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ycycse ycycse ycycse approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@HTHou @ycycse @rmpinchback