Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modify the result json format of mini load #1487

Merged
merged 6 commits into from
Jul 16, 2019
Merged

Modify the result json format of mini load #1487

merged 6 commits into from
Jul 16, 2019

Conversation

morningman
Copy link
Contributor

Mini load is now using stream load framework. But we should keep the
mini load return behavior and result json format be same as old.
So PUBLISH_TIMEOUT error should be treated as OK in mini load.

Also add 2 counters for OlapTableSink profile:
SerializeBatchTime: time of serializing all row batch.
WaitInFlightPacketTime: time of waiting last send packet

@morningman
Modify the result json format of mini load.
c1f1479 
Mini load is now using stream load framework. But we should keep the
mini load return behavior and result json format be same as old.
So PUBLISH_TIMEOUT error should be treated as OK in mini load.
@morningman
remove txn id in mini load result
24b504d
@morningman
add wait_in_flight_packet_timer
2bcbdd5
@morningman
fix misspell
40a736d
@morningman
remove label_alread_used case in mini load resutl to_json() method
a655178
imay
imay reviewed Jul 16, 2019
View reviewed changes
be/src/exec/tablet_sink.h Outdated
@@ -168,6 +168,9 @@ class OlapTableSink : public DataSink {
return _profile;
}

int64_t* wait_in_flight_packet_ns() { return &_wait_in_flight_packet_ns; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
int64_t* wait_in_flight_packet_ns() { return &_wait_in_flight_packet_ns; }
int64_t* mutable_wait_in_flight_packet_ns() { return &_wait_in_flight_packet_ns; }

be/src/exec/tablet_sink.h Outdated
@@ -168,6 +168,9 @@ class OlapTableSink : public DataSink {
return _profile;
}

int64_t* wait_in_flight_packet_ns() { return &_wait_in_flight_packet_ns; }
int64_t* serialize_batch_ns() { return &_serialize_batch_ns; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
int64_t* serialize_batch_ns() { return &_serialize_batch_ns; }
int64_t* mutable_serialize_batch_ns() { return &_serialize_batch_ns; }

imay
imay approved these changes Jul 16, 2019
View reviewed changes
Copy link
Contributor

@imay imay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 4e043e6 into apache:master Jul 16, 2019
morningman added a commit to baidu-doris/incubator-doris that referenced this pull request Jul 16, 2019
@morningman
Modify the result json format of mini load (apache#1487)
0f0cc14 
Mini load is now using stream load framework. But we should keep the
mini load return behavior and result json format be same as old.
So PUBLISH_TIMEOUT error should be treated as OK in mini load.

Also add 2 counters for OlapTableSink profile:
SerializeBatchTime: time of serializing all row batch.
WaitInFlightPacketTime: time of waiting last send packet
HangyuanLiu added a commit to HangyuanLiu/incubator-doris that referenced this pull request Jul 17, 2019
@HangyuanLiu
fix
af9ce89 
fix

fix

fix

add

FixBug: if columns of doris table less than parquet file columns , BE will be crash (apache#1464)

Build snappy with optimize-options enabled (apache#1467)

Fix bug when use SELECT * FROM TABLE LIMIT 1 (apache#1469)

Refactor Storage Engine (apache#1478)

NOTE: This patch would modify all Backend's data.
And this will cause a very long time to restart be.
So if you want to interferer your product environment,
you should upgrade backend one by one.

1. Refactoring be is to clarify the structure the codes.
2. Use unique id to indicate a rowset.
   Nameing rowset with tablet_id and version will lead to
   many conflicts among compaction, clone, restore.
3. Extract an rowset interface to encapsulate rowsets
   with different format.

Remove unused code (apache#1483)

Add timeout in stream load planner (apache#1480)

Mini load timeout needs to be added in plan options.
The timeout property has been added in request of process put.
Otherwise, the timeout of mini load is useless.

Add log of label, txn and query id in mini load

Fix heap-buffer-overflow in split_part() function in StringFunctions (apache#1482)

fix

fix

fix

add

fix

add

fix

fix

fix

fix

fix

fix

fix

fix

fix

fix

fix

fix

fix

fix

fix

Modify the result json format of mini load (apache#1487)

Mini load is now using stream load framework. But we should keep the
mini load return behavior and result json format be same as old.
So PUBLISH_TIMEOUT error should be treated as OK in mini load.

Also add 2 counters for OlapTableSink profile:
SerializeBatchTime: time of serializing all row batch.
WaitInFlightPacketTime: time of waiting last send packet

 Support grant GRANT_PRIV on database or table level (apache#1472)

Currently, GRANT_PRIV can only be granted on global level, which means
it can only be granted on *.*. Grant it on db.* or db.tbl are not allowed.

This will not be able to meet the requirement to create a user who has privilege
to grant privileges to other users on specified database or table, such as:

GRANT SELECT_PRIV ON db1.* TO cmy@'%';

So I extend the range of GRANT_PRIV. User can now grant GRANT_PRIV on
database or even table level, such as:

GRANT GRANT_PRIV ON db1.* TO cmy@'%';

And after being granted, the user cmy@'%' can now grant GRANT_PRIV on db1.* to
other users.

Add partition id to tablet meta in be (apache#1490)

FE uses partition_id to publish version. BE should check whether all tablets related with this partition have the version. But Tablet in BE does not have partition id in its metadata. So that BE could not check it.

This patch will add partition id to tablet meta during report task.
Sync at most 10k tablets during set tablet meta.

fix

fix

fix

fix

fix

fix

fix

fix

fix

add
@imay imay mentioned this pull request Sep 26, 2019
Closed
luwei16 pushed a commit to luwei16/incubator-doris that referenced this pull request Apr 7, 2023
@platoneko
[feature](selectdb-cloud) Implement recycle copy jobs http api (apach…
98c6f38 
…e#1487)

Usage:
```
GET /RecyclerService/http/recycle_copy_jobs?token={token}&instance_id={instance_id} HTTP/1.1
Content-Length: <ContentLength>
Content-Type: text/plain
```
e.g.

```
curl 127.0.0.1:5100/RecyclerService/http/recycle_copy_jobs?token={token}&instance_id={instance_id}
```
SWJTU-ZhangLei pushed a commit to SWJTU-ZhangLei/incubator-doris that referenced this pull request Jul 25, 2023
@platoneko @gavinchou
[feature](selectdb-cloud) Implement recycle copy jobs http api (apach…
001e81c 
…e#1487)

Usage:
```
GET /RecyclerService/http/recycle_copy_jobs?token={token}&instance_id={instance_id} HTTP/1.1
Content-Length: <ContentLength>
Content-Type: text/plain
```
e.g.

```
curl 127.0.0.1:5100/RecyclerService/http/recycle_copy_jobs?token={token}&instance_id={instance_id}
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@imay imay imay approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@morningman @imay