Clean base64 #4
Closed
Clean base64 #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Joe-Abraham
force-pushed
the
clean_base64
branch
5 times, most recently
from
0cfc65d
to
138718c
Compare
Summary: `SIME` --> `SIMD` Pull Request resolved: facebookincubator#10319 Reviewed By: amitkdutta Differential Revision: D59122175 Pulled By: bikramSingh91 fbshipit-source-id: 63a56176df0528ea5217cad903e48ebb120a8551
) Summary: Pull Request resolved: facebookincubator#10304 Reviewed By: amitkdutta Differential Revision: D59122023 Pulled By: bikramSingh91 fbshipit-source-id: ec8c567cfc47cf9f562672ae4fdcc47f58d4d06e
…tor#9848) Summary: Spark mod support float/double types (https://github.com/apache/spark/blob/403619a3974c595ba80d6c9dbd23b8c2f1e2233e/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala#L899) Pull Request resolved: facebookincubator#9848 Reviewed By: xiaoxmeng Differential Revision: D59122289 Pulled By: bikramSingh91 fbshipit-source-id: 1eb00a504b05ce74fb4ee405464eec88552b4feb
…tor#9992) Summary: Gluten project needs to pre-build arrow before building velox. And it's also possible that arrow libs have already been installed, e.g., via vcpkg. Then, velox doesn't need to build arrow from source again. Pull Request resolved: facebookincubator#9992 Reviewed By: xiaoxmeng Differential Revision: D59122226 Pulled By: bikramSingh91 fbshipit-source-id: 8a0320cc2a57cec3bf3aa894ccc1e8f8c1fb175b
Summary: This change adds compatibility with newer versions of Protobuf. The actual version of Protobuf is not being updated. Protobuf v27.2 was used for testing. Protobuf v21.8 is still supported. Fixes facebookincubator#10133 Pull Request resolved: facebookincubator#10294 Reviewed By: xiaoxmeng Differential Revision: D59122442 Pulled By: bikramSingh91 fbshipit-source-id: d6f92a30361d32eae0189e7b2c17fbb592c75925
Summary: At present, RowNumber only supports single-level spill partitions. Once a spill takes place, it won't be spilled again. This means that even if a restored spill partition uses an excessive amount of memory, it still cannot be reclaimed. Therefore, it would be beneficial to implement recursive spill support, as outlined in https://facebookincubator.github.io/velox/develop/spilling.html. 1 Advance 'spillPartitionBits_' based on the partition bit of the currently restoring spill partition to prepare for potential subsequent spills (recursive). 2 During the recursive spill input process, read the spilled input data from the previous spill run via 'spillInputReader_', then spill it back into several sub-partitions. Then restore one of the newly spilled partitions and reset 'spillInputReader' accordingly. 3 It's crucial to separate the recursive input spill process from the spill process itself, as it can be time-consuming and should yield if it exceeds the CPU time limit. In summary, the first spill operation writes all data to the disk. Subsequently, RowNumber reads the data from the disk one partition at a time and clears the data from the restored partition once it has been processed (facebookincubator#8413). In the case of a recursive spill, only the data from the currently restored partition in memory is spilled to the disk (next level), and only one partition from the next level is loaded back into memory. The data layout may look like as follows. ||Memory|Disk| | -- | -- | -- | |No-Spill | [P0, P1]| []| |First Spill | []| [P0, P1]| |Restore | [P0] | [P1]| |Second Spill | [] | [P0-0, P0-1, P1]| |Restore| [P0-0] | [P0-1, P1]| |Processed |[] |[P0-1, P1]| |Restored |[P0-1] |[P1]| |Processed | [] | [P1]| |Restored | [P1] | []| Pull Request resolved: facebookincubator#8654 Reviewed By: oerling Differential Revision: D54493422 Pulled By: xiaoxmeng fbshipit-source-id: 40dc02180d7fce46131fe98f68a00a776caceccc
…tor#10260) Summary: This PR solves the following issues: 1. `cumulativeBytes_` is the counter of allocated bytes, if we allocate n blocks and then execute free() on these blocks, the final value of `cumulativeBytes_` should be equal to 0. But without this PR, the value of `cumulativeBytes_` is not equal to 0. see `HashStringAllocatorTest#multipleFreeAncCheckCumulativeBytes` test case in this PR. This is because we don't include the size of the `Header` when calculating the allocated bytes. 2. When executing `HashStringAllocator::clear()`, we need to decrement memory counters. PR facebookincubator#10053 only decrement memory counters when releasing `allocationsFromPool_`. We also need to decrement the memory counters when releasing memory requested by `memory::AllocationPool`. 3. After executing `HashStringAllocator::clear()`, the values of `cumulativeBytes_` and `sizeFromPool_` should be equal to 0. CC: xiaoxmeng mbasmanova Pull Request resolved: facebookincubator#10260 Reviewed By: Yuhta Differential Revision: D59065411 Pulled By: xiaoxmeng fbshipit-source-id: 8f7913b0b89de69f88fa4d606c9628b31be38e3a
Summary: The semantics of the right join are similar to the left join, so we referenced the implementation of the left join to achieve the implementation of the right join. Pull Request resolved: facebookincubator#10148 Reviewed By: bikramSingh91 Differential Revision: D59176120 Pulled By: pedroerp fbshipit-source-id: 95184725dfa5fea9317c822d7761507bc49fca9b
Summary: Pull Request resolved: facebookincubator#10344 There is a lot more clean up to be done in this file, this just targets a single test fixture. Velox functions now support the logical type TimeStampWithTimeZone. The use of "evaluateWithTimestampWithTimezone" is therefore no longer needed. Reviewed By: pedroerp Differential Revision: D59135897 fbshipit-source-id: e1ccf1e65d7c95c2d3d4be7b2346ac398876836a
Summary:
boost::lexical_cast used to implement cast(uuid as varchar) is very slow.
Replace with custom optimization.
Microbenchmark shows 20x improvement.
The benchmark compares uuid() with cast(uuid() as varchar). The latter includes the code of the former plus the cost of the cast. Before the change, uuid() + cast was 16s, 10s more than uuid() alone. After the change, uuid() + cast() is just 0.5s more.
Before:
```
============================================================================
[...]hmarks/ExpressionBenchmarkBuilder.cpp relative time/iter iters/s
============================================================================
cast##no_cast 6.33s 157.86m
cast##as_varchar 16.54s 60.45m
```
After:
```
============================================================================
cast##no_cast 6.29s 159.10m
cast##as_varchar 6.81s 146.74m
----------------------------------------------------------------------------
```
Profile before the optimization: {F1735048022}
Profile after the optimization: {F1735104992}
Reviewed By: xiaoxmeng
Differential Revision: D59229653
fbshipit-source-id: 1772b00e13109d49f829efb36fe706e511a5ed25
Summary: Pull Request resolved: facebookincubator#10350 Reviewed By: tanjialiang Differential Revision: D59232726 Pulled By: xiaoxmeng fbshipit-source-id: 663ff3163460adddd5b55a46e09996cc07d4fc83
Summary: This is a temporary workaround. The actual fix is to build and install thrift. Resolves facebookincubator#10352 Pull Request resolved: facebookincubator#10360 Reviewed By: amitkdutta Differential Revision: D59239687 Pulled By: pedroerp fbshipit-source-id: 733268cab0742444e6fd637854b711f918308299
…acebookincubator#9295) Summary: facebookincubator#5762 prestodb/presto#20530 Pull Request resolved: facebookincubator#9295 Reviewed By: amitkdutta Differential Revision: D58423391 Pulled By: kevinwilfong fbshipit-source-id: 93d30d872b5a1924c86cb1f2c2d4c057ee91d02a
Summary: Add a max spill level config in the RowNumberFuzzer. Pull Request resolved: facebookincubator#10347 Reviewed By: tanjialiang Differential Revision: D59232589 Pulled By: xiaoxmeng fbshipit-source-id: 95a2d37d387c5767cb61444d37a154d936571a2b
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.