Configurable data source for offloaded messages #8717

Renkai · 2020-11-26T06:43:20Z

Fix issue: #8591

This PR include:

API change in command tools
Related implementation with tests
Related docs in cookbook

By the way log4j dependency is removed for module managed-ledger because now the whole project use log4j2 as the default logger framework.

Renkai · 2020-11-26T09:06:55Z

/pulsarbot run-failure-checks

gaoran10 · 2020-11-30T17:10:16Z

Great work, maybe we could add a configuration like tieredStorageReadPriority at the config file broker.conf or standalone.conf for broker level configuration.

Renkai · 2020-12-02T08:29:39Z

Great work, maybe we could add a configuration like tieredStorageReadPriority at the config file broker.conf or standalone.conf for broker level configuration.
@gaoran10
Now added to broker.conf

Renkai · 2020-12-02T13:34:07Z

/pulsarbot run-failure-checks

Signed-off-by: Renkai <gaelookair@gmail.com>

Renkai · 2020-12-03T05:24:02Z

/pulsarbot run-failure-checks

--- *Motivation* The pakcage management module build failed by the spotbugs check. That will block the project build.

Signed-off-by: Renkai <gaelookair@gmail.com>

Renkai · 2020-12-03T12:01:15Z

/pulsarbot run-failure-checks

Renkai · 2020-12-03T12:55:51Z

/pulsarbot run-failure-checks

Renkai · 2020-12-04T00:04:53Z

/pulsarbot run-failure-checks

Signed-off-by: Renkai <gaelookair@gmail.com>

sijie · 2021-01-04T17:43:11Z

@hangc0276 Can you review this PR again?

hangc0276 · 2021-01-05T12:21:29Z

@hangc0276 Can you review this PR again?

@sijie done

# Conflicts: # managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java

Signed-off-by: Renkai <gaelookair@gmail.com>

sijie

@Renkai Overall the change looks great. I have left a couple of comments. Please take a look.

sijie · 2021-01-06T06:17:02Z

pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopics.java

+            if (this.offloadReadPriorityStr != null) {
+                try {
+                    offloadedReadPriority = OffloadedReadPriority.fromString(this.offloadReadPriorityStr);
+                    System.out.println("offloadedReadPriority = " + offloadedReadPriority);


@Renkai Can we avoid using System.out.println?

@sijie This code is in a command-line tool, I think stdout is a good choice for users to get information

This is a set command, no?

@sijie I check the code again, I think it's ok to remove println so we can use mute as success information, if the value which user set is an invalid value, the tool will throw a exception. The println was removed.

sijie · 2021-01-06T06:17:06Z

pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdNamespaces.java

+            if (this.offloadReadPriorityStr != null) {
+                try {
+                    offloadedReadPriority = OffloadedReadPriority.fromString(this.offloadReadPriorityStr);
+                    System.out.println("offloadedReadPriority = " + offloadedReadPriority);


@Renkai Can we avoid using System.out.println?

But this is a set command, no? Why people need this information when setting the offloaded priority?

pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/OffloadPolicies.java

Signed-off-by: Renkai <gaelookair@gmail.com>

sijie · 2021-01-06T07:45:43Z

pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdNamespaces.java

+            if (this.offloadReadPriorityStr != null) {
+                try {
+                    offloadedReadPriority = OffloadedReadPriority.fromString(this.offloadReadPriorityStr);
+                    System.out.println("offloadedReadPriority = " + offloadedReadPriority);


But this is a set command, no? Why people need this information when setting the offloaded priority?

sijie · 2021-01-06T07:45:54Z

pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopics.java

+            if (this.offloadReadPriorityStr != null) {
+                try {
+                    offloadedReadPriority = OffloadedReadPriority.fromString(this.offloadReadPriorityStr);
+                    System.out.println("offloadedReadPriority = " + offloadedReadPriority);


This is a set command, no?

Signed-off-by: Renkai <gaelookair@gmail.com>

sijie · 2021-01-08T05:01:23Z

Great contributions!

Fix issue: #8591 This PR include: * API change in command tools * Related implementation with tests * Related docs in cookbook By the way log4j dependency is removed for module `managed-ledger` because now the whole project use log4j2 as the default logger framework. (cherry picked from commit 7c09f5c)

Anonymitaet · 2021-02-25T07:26:49Z

@Renkai many thanks for your docs! While site2/docs/cookbooks-tiered-storage.md is deprecated. We will fix it.

Anonymitaet · 2021-03-04T06:02:50Z

@Renkai I've submitted #9795 to add docs, could you please help review? Thanks

Renkai marked this pull request as ready for review November 26, 2020 07:09

Renkai mentioned this pull request Nov 26, 2020

Add data source priority configuration for offloaded messages #8678

Closed

Renkai force-pushed the read_priority_full branch from 2d14cec to 6a071f7 Compare December 2, 2020 03:05

Renkai added 13 commits December 3, 2020 08:30

add configuration for read priority

2b654e8

Signed-off-by: Renkai <gaelookair@gmail.com>

add read priority to offload policy

7f8ebff

Signed-off-by: Renkai <gaelookair@gmail.com>

add read priority to offload policy

7a6942d

Signed-off-by: Renkai <gaelookair@gmail.com>

add read priority to offload policy

ec43abe

Signed-off-by: Renkai <gaelookair@gmail.com>

add read priority to offload policy

9cd6b87

Signed-off-by: Renkai <gaelookair@gmail.com>

for bookkeeper first read priority read bookkeeper first

a697370

Signed-off-by: Renkai <gaelookair@gmail.com>

nightly commit

61a9561

Signed-off-by: Renkai <gaelookair@gmail.com>

drop log pom

969d164

Signed-off-by: Renkai <gaelookair@gmail.com>

add test

0ba829c

Signed-off-by: Renkai <gaelookair@gmail.com>

add test

d70bd00

Signed-off-by: Renkai <gaelookair@gmail.com>

add test

a1031a6

Signed-off-by: Renkai <gaelookair@gmail.com>

add broker conf managedLedgerDataReadPriority

fd224e1

Signed-off-by: Renkai <gaelookair@gmail.com>

read broker configuration

1cb61da

Signed-off-by: Renkai <gaelookair@gmail.com>

Renkai force-pushed the read_priority_full branch from 578727a to 1cb61da Compare December 3, 2020 00:30

zymap and others added 3 commits December 3, 2020 14:36

The package management module failed to build

d3148f4

--- *Motivation* The pakcage management module build failed by the spotbugs check. That will block the project build.

bug fix

7ee26cb

Signed-off-by: Renkai <gaelookair@gmail.com>

Merge branch 'fix-spotbugs-issue' into read_priority_full

1fe3a3d

codelipenghui assigned Renkai Dec 3, 2020

codelipenghui added this to the 2.8.0 milestone Dec 3, 2020

Renkai added 4 commits December 27, 2020 15:44

small fix

d0505bf

Signed-off-by: Renkai <gaelookair@gmail.com>

small fix

32658b6

Signed-off-by: Renkai <gaelookair@gmail.com>

small fix

a486080

Signed-off-by: Renkai <gaelookair@gmail.com>

Merge branch 'master' into read_priority_full

4d22a47

hangc0276 approved these changes Jan 5, 2021

View reviewed changes

Renkai added 3 commits January 6, 2021 09:16

Merge branch 'master' into read_priority_full

9c99302

# Conflicts: # managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java

fix conflict

142e1ed

Signed-off-by: Renkai <gaelookair@gmail.com>

change driver names to immutable for bugspots

abf10ab

Signed-off-by: Renkai <gaelookair@gmail.com>

sijie reviewed Jan 6, 2021

View reviewed changes

Renkai added 2 commits January 6, 2021 15:25

rename to tiered storage first

bbd54c8

Signed-off-by: Renkai <gaelookair@gmail.com>

rename to tiered storage first

7506000

Signed-off-by: Renkai <gaelookair@gmail.com>

sijie requested changes Jan 6, 2021

View reviewed changes

rename to tiered storage first

280715c

Signed-off-by: Renkai <gaelookair@gmail.com>

Renkai requested a review from sijie January 6, 2021 08:47

sijie added area/tieredstorage doc-required Your PR changes impact docs and you will update later. labels Jan 8, 2021

sijie approved these changes Jan 8, 2021

View reviewed changes

sijie merged commit 7c09f5c into apache:master Jan 8, 2021

codelipenghui mentioned this pull request Jan 19, 2021

ISSUE-8591: Configurable data source for offloaded messages streamnative/pulsar-archived#2022

Closed

4 tasks

codelipenghui added release/2.7.1 cherry-picked/branch-2.7 Archived: 2.7 is end of life labels Jan 31, 2021

Anonymitaet mentioned this pull request Mar 4, 2021

[Doc] Add descriptions for set-offload-policies #9795

Merged

Anonymitaet removed the doc-required Your PR changes impact docs and you will update later. label Mar 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configurable data source for offloaded messages #8717

Configurable data source for offloaded messages #8717

Renkai commented Nov 26, 2020 •

edited

Loading

Renkai commented Nov 26, 2020

gaoran10 commented Nov 30, 2020

Renkai commented Dec 2, 2020

Renkai commented Dec 2, 2020

Renkai commented Dec 3, 2020

Renkai commented Dec 3, 2020

Renkai commented Dec 3, 2020

Renkai commented Dec 4, 2020

sijie commented Jan 4, 2021

hangc0276 commented Jan 5, 2021

sijie left a comment

sijie Jan 6, 2021

Renkai Jan 6, 2021

sijie Jan 6, 2021

Renkai Jan 6, 2021

sijie Jan 6, 2021

Renkai Jan 6, 2021

sijie Jan 6, 2021

Renkai Jan 6, 2021

sijie Jan 6, 2021

sijie Jan 6, 2021

sijie commented Jan 8, 2021

Anonymitaet commented Feb 25, 2021

Anonymitaet commented Mar 4, 2021

Configurable data source for offloaded messages #8717

Configurable data source for offloaded messages #8717

Conversation

Renkai commented Nov 26, 2020 • edited Loading

Renkai commented Nov 26, 2020

gaoran10 commented Nov 30, 2020

Renkai commented Dec 2, 2020

Renkai commented Dec 2, 2020

Renkai commented Dec 3, 2020

Renkai commented Dec 3, 2020

Renkai commented Dec 3, 2020

Renkai commented Dec 4, 2020

sijie commented Jan 4, 2021

hangc0276 commented Jan 5, 2021

sijie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sijie commented Jan 8, 2021

Anonymitaet commented Feb 25, 2021

Anonymitaet commented Mar 4, 2021

Renkai commented Nov 26, 2020 •

edited

Loading