Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revisit publishing of uncommited messages in functional_test #1733

Closed
dnwe opened this issue Jun 24, 2020 · 2 comments · Fixed by #1939
Closed

Revisit publishing of uncommited messages in functional_test #1733

dnwe opened this issue Jun 24, 2020 · 2 comments · Fixed by #1939

Comments

@dnwe
Copy link
Collaborator

dnwe commented Jun 24, 2020

One of the pieces of #1695 was to use a small jar for publishing uncommited messages as part of the functional test:

https://github.com/Shopify/sarama/blob/221ed1a25150fa98d750f2e2f33f58ff018238b4/functional_test.go#L343-L345

What's missing from #1152 to allow us to do this in-house?

/cc @KJTsanaktsidis

@KJTsanaktsidis
Copy link
Contributor

So I had originally thought that there was just no support for the nescessary calls to do transactional publishing at all, but actually the low-level support in the Broker/Protocol interface does exist, it's just not part of the high-level producer interface.

Just purely for the purpose of getting rid of the JAR, I wonder if something like this might work:

	transactionalID := "transactional.id config"
	coordRes, _ := controller.FindCoordinator(&FindCoordinatorRequest{
		Version: 2,
		CoordinatorKey: transactionalID,
		CoordinatorType: CoordinatorTransaction,
	})
	pidRes, _ := coordRes.Coordinator.InitProducerID(&InitProducerIDRequest{
		TransactionalID: &transactionalID,
		TransactionTimeout: 10 * time.Second,
	})
	_, _ = coordRes.Coordinator.AddPartitionsToTxn(&AddPartitionsToTxnRequest{
		TransactionalID: transactionalID,
		ProducerID: pidRes.ProducerID,
		ProducerEpoch: pidRes.ProducerEpoch,
		TopicPartitions: map[string][]int32{
			"uncomitted-topic-test-4": {0},
		},
	})
	ps := &produceSet{
		msgs:          make(map[string]map[int32]*partitionSet),
		parent:        &asyncProducer{
			conf: config,
		},
		producerID:    pidRes.ProducerID,
		producerEpoch: pidRes.ProducerEpoch,
	}
	_ = ps.add(&ProducerMessage{
		Topic: "uncomitted-topic-test-4",
		Partition: 0,
		Value: StringEncoder("uncomimtted message"),
	})
	produceReq := ps.buildRequest()
	produceReq.TransactionalID = &transactionalID
	_, _ = coordRes.Coordinator.Produce(produceReq)
	_, _ = coordRes.Coordinator.EndTxn(&EndTxnRequest{
		TransactionalID: transactionalID,
		ProducerID: pidRes.ProducerID,
		ProducerEpoch: pidRes.ProducerEpoch,
		TransactionResult: false,
	})

I haven't actually tried to run it but I can give it a shot when I get a moment and see how it goes.

@ghost
Copy link

ghost commented Mar 16, 2021

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the master branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

@ghost ghost added the stale Issues and pull requests without any recent activity label Mar 16, 2021
@dnwe dnwe removed the stale Issues and pull requests without any recent activity label May 7, 2021
dnwe added a commit that referenced this issue May 7, 2021
One of the pieces of #1695 was to use a small jar for publishing
uncommited messages as part of the functional test. Replacing that with
the native Sarama transactional producer

Fixes #1733

Co-authored-by: KJTsanaktsidis <ktsanaktsidis@zendesk.com>
dnwe added a commit that referenced this issue Sep 13, 2021
One of the pieces of #1695 was to use a small jar for publishing
uncommited messages as part of the functional test. Replacing that with
the native Sarama transactional producer

Fixes #1733

Co-authored-by: KJTsanaktsidis <ktsanaktsidis@zendesk.com>
dnwe added a commit that referenced this issue Feb 8, 2022
One of the pieces of #1695 was to use a small jar for publishing
uncommited messages as part of the functional test. Replacing that with
the native Sarama transactional producer

Fixes #1733

Co-authored-by: KJTsanaktsidis <ktsanaktsidis@zendesk.com>
dnwe added a commit that referenced this issue Feb 8, 2022
One of the pieces of #1695 was to use a small jar for publishing
uncommited messages as part of the functional test. Replacing that with
the native Sarama transactional producer

Fixes #1733

Co-authored-by: KJTsanaktsidis <ktsanaktsidis@zendesk.com>
dnwe added a commit that referenced this issue Feb 8, 2022
One of the pieces of #1695 was to use a small jar for publishing
uncommited messages as part of the functional test. Replacing that with
the native Sarama transactional producer

Fixes #1733

Co-authored-by: KJTsanaktsidis <ktsanaktsidis@zendesk.com>
dnwe added a commit that referenced this issue Feb 8, 2022
One of the pieces of #1695 was to use a small jar for publishing
uncommited messages as part of the functional test. Replacing that with
the native Sarama transactional producer

Fixes #1733

Co-authored-by: KJTsanaktsidis <ktsanaktsidis@zendesk.com>
@dnwe dnwe closed this as completed in #1939 Feb 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants