Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-406: [C++] Set explicit 64K HDFS buffer size, test large reads #226

Closed
wants to merge 2 commits into from
Closed

ARROW-406: [C++] Set explicit 64K HDFS buffer size, test large reads #226

wants to merge 2 commits into from

Conversation

wesm
Copy link
Member

@wesm wesm commented Dec 5, 2016

We could not support reads in excess of the default buffer size (typically 64K)

@wesm
Set explicit 64K HDFS buffer size, test large reads using buffered ch…
0028e90 
…unks

Change-Id: Ie59456c763538956520c7f606717bcc6a99c9af1
xhochy
xhochy approved these changes Dec 6, 2016
View reviewed changes
Copy link
Member

@xhochy xhochy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM besides the clang-tidy warning

@wesm
cpplint
d09b645 
Change-Id: Idbbc394fd12f5d93d889b0b2ec669d9030f69770
@wesm
Copy link
Member Author

wesm commented Dec 6, 2016

+1

@wesm wesm changed the title ARROW-406: Set explicit 64K HDFS buffer size, test large reads ARROW-406: [C++] Set explicit 64K HDFS buffer size, test large reads Dec 6, 2016
@asfgit asfgit closed this in 4b72329 Dec 6, 2016
@wesm wesm deleted the ARROW-406 branch December 6, 2016 16:52
wesm added a commit to wesm/arrow that referenced this pull request Sep 2, 2018
@wesm
PARQUET-842: Do not set unnecessary fields in the parquet::SchemaElement
d2f317e 
The Impala Parquet scanner can reject our data otherwise

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes apache#226 from wesm/PARQUET-842 and squashes the following commits:

0cfa53c [Wes McKinney] Do not set field_id in SchemaElement until we understand why it is causing parquet-mr problems
4220b48 [Wes McKinney] Do not set unnecessary fields in the parquet::SchemaElement
wesm added a commit to wesm/arrow that referenced this pull request Sep 4, 2018
@wesm @xhochy
PARQUET-842: Do not set unnecessary fields in the parquet::SchemaElement
28f182e 
The Impala Parquet scanner can reject our data otherwise

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes apache#226 from wesm/PARQUET-842 and squashes the following commits:

0cfa53c [Wes McKinney] Do not set field_id in SchemaElement until we understand why it is causing parquet-mr problems
4220b48 [Wes McKinney] Do not set unnecessary fields in the parquet::SchemaElement

Change-Id: I2a7902765e92f0da386e9e65513dd0954113fa60
wesm added a commit to wesm/arrow that referenced this pull request Sep 6, 2018
@wesm @xhochy
PARQUET-842: Do not set unnecessary fields in the parquet::SchemaElement
bcaa02e 
The Impala Parquet scanner can reject our data otherwise

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes apache#226 from wesm/PARQUET-842 and squashes the following commits:

0cfa53c [Wes McKinney] Do not set field_id in SchemaElement until we understand why it is causing parquet-mr problems
4220b48 [Wes McKinney] Do not set unnecessary fields in the parquet::SchemaElement

Change-Id: I2a7902765e92f0da386e9e65513dd0954113fa60
wesm added a commit to wesm/arrow that referenced this pull request Sep 7, 2018
@wesm @xhochy
PARQUET-842: Do not set unnecessary fields in the parquet::SchemaElement
85a14bb 
The Impala Parquet scanner can reject our data otherwise

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes apache#226 from wesm/PARQUET-842 and squashes the following commits:

0cfa53c [Wes McKinney] Do not set field_id in SchemaElement until we understand why it is causing parquet-mr problems
4220b48 [Wes McKinney] Do not set unnecessary fields in the parquet::SchemaElement

Change-Id: I2a7902765e92f0da386e9e65513dd0954113fa60
wesm added a commit to wesm/arrow that referenced this pull request Sep 8, 2018
@wesm @xhochy
PARQUET-842: Do not set unnecessary fields in the parquet::SchemaElement
493603d 
The Impala Parquet scanner can reject our data otherwise

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes apache#226 from wesm/PARQUET-842 and squashes the following commits:

0cfa53c [Wes McKinney] Do not set field_id in SchemaElement until we understand why it is causing parquet-mr problems
4220b48 [Wes McKinney] Do not set unnecessary fields in the parquet::SchemaElement

Change-Id: I2a7902765e92f0da386e9e65513dd0954113fa60
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Dec 14, 2021
@rafael-telles
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
a38381b 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
rafael-telles added a commit to rafael-telles/arrow that referenced this pull request Jan 20, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
743ce3a 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Mar 29, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
b258fe6 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Mar 29, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
20be27f 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Mar 29, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
25bcd99 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Mar 29, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
d503cb1 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Mar 29, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
d610d51 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Mar 29, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
8b06d38 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Apr 4, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
2e2a84c 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Apr 19, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
1e8bcff 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Apr 19, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
0acb3b4 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Apr 20, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
5867928 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Apr 20, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
1b3c73e 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Apr 20, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
7fe68e2 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Apr 20, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
933bd58 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Apr 20, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
e8e7512 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
vfraga pushed a commit to rafael-telles/arrow that referenced this pull request Apr 20, 2022
@rafael-telles @vfraga
Add missing metadata on Arrow schemas returned by Flight SQL's GetTab…
877d235 
…les and query execution methods. (apache#226)

This add an auxiliary class FlightSqlColumnMetadata meant to read and write known metadata for Arrow schema fields, such as CATALOG_NAME, SCHEMA_NAME, TABLE_NAME, PRECISION, SCALE, IS_AUTO_INCREMENT, IS_CASE_SENSITIVE, IS_READ_ONLY and IS_SEARCHABLE.
@paleolimbot paleolimbot mentioned this pull request Jan 28, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xhochy xhochy xhochy approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@wesm @xhochy