https://gbraidfinder.ogres.cc/home
Designed to get more information from a raid than what was tweeted.
- gbdist - server that listens to all raids and updates and accepts socket connections from users (Nodejs, Nestjs, socket.io, rxjs, kafkajs)
- gbr - A node microservice that runs puppeteer and has a logged in gbf account active and waiting to get info about a tweeted raid (Nodejs, Nestjs, rxjs, kafkajs)
- gbsource - server that gets tweets and sends them to redpanda (Nodejs, Nestjs, rxjs, kafkajs)
- k8s - kubernetes helm chart configuration for deployment (Kubernetes, Helm)
- redpanda - Helm chart for redpanda, taken from https://github.com/redpanda-data/helm-charts/tree/36cb1a3169c14f396924a7942dc4cfd07cfdbb22/charts/redpanda
Backend gets each tweet and passes it to the user and to a cluster of alt accounts at the same time. The backend chooses the account with the smallest queue. The logged in alt accounts make requests to get information about a raid. Sends that info to everyone subscribed to updates for that raid.
Uses Twitter API Filtered Stream v2 to get tweets
You can deploy your own version of this, free if you use minikube for a local deployment. However, it's kinda pointless if you don't have alt accounts to get the updated info.
There's still alot that could be updated, the code isn't in the best state because the architecture went through a couple iterations and I didn't clean up the code.
- Fork the repo
- Clone it to a local directory
- Create a branch
- Make any neccessary changes (see "Running the Project Locally" Section)
- Push changes to your forked repo
- Open a pull request from your forked repo for review
For more information about contributing for the first time see here: https://github.com/firstcontributions/first-contributions/blob/main/README.md
The frontend of this project can be modified fairly easily by anyone. You'll need to have npm and nodejs installed (preferably at latest stable version)
Open the project and cd into the front
folder.
Run npm install
to install dependencies.
Run npm start
to host a local version of the project at localhost:4200
Currently this will connect you to the backend that is already running without need to run the backend locally. (This may change in the future)
To run the backend locally, you have the option to use minikube to run a local kubernetes cluster and deploy everything there using helm.
-
For this option you will need to install docker, minikube, and helm
-
Create a 'values.yaml' in the k8s directory and fill in at least 1 twitter api tokens, and add any account information you want to use. Use the values.example.yaml as a template.
-
Note: You may also need to modify the
gbdist-service
template ink8s/templates/gbdist-deployment.yaml
-
Install redpanda or kafka in your minikube cluster.
-
Install the helm chart for this project by going into the
k8s
directory and runninghelm install .
-
To access chrome and log in to the accounts run
kubectl port-forward "deploy/gbr-0" 9222:9222
(incrementing values for multiple accounts ex.kubectl port-forward "deploy/gbr-1" 9223:9222
) -
Then visit chrome://inspect/#devices and Configure as many additional ports you need. After a couple seconds it should allow you to inspect the page and log in.
-
Reccomendations: I reccomend running the following in your k8s cluster:
- weavescope for monitoring pods more efficiently, accessing logs, and shells
- redpanda-console for monitoring redpanda, and checking which topics have been assigned to which topic partition
- To install default console chart with specific brokers you can use the following command:
helm install -n redpanda redpanda-console redpanda-console/console --set-json 'console.config.kafka.brokers=["broker1:9092", "broker2:9093"]'
- To install default console chart with specific brokers you can use the following command:
-
Run redpanda locally, you can use
redpanda/docker-compose.yaml
to run a cluster of three redpanda instances locally. -
Update any config files for apps you want to run (ex.
gbr/config/config.json
) -
For running gbr, you can also modify any credentials in
gbr/package.json
. -
Alternatively you can also remove them and export the configs in your shell if running multiple instances.
-
Run
npm start
to start all of the applications.
Since each raid requires a minimum level to get updated info, each raid can't be distributed to every account. Therefore we create a topic for each minimum rank requirement (ex. l101, l120, l150, l151). From there accounts only get raids from topics that are less than or equal to their rank. For example a rank 160 account will get raids from l151, l150, l130, l120,... etc. But won't get raids from l170 or l200.
However, we also want to evenly distribute the raids so that one account isn't handling all of the raids for a topic. The way this is accomplished is by creating the same amount or more partitions as accounts. Raids per topic are distributed evenly per partition. For example if raid A, B, C are sent to topic l101, raid A will be sent to partition 0, raid B will be sent to partition 1, and raid C will be sent to partition 2.
From there we can assign each partition to a gbr account seperately. This is possible when the groupid of every gbr is the same. A group leader is chosen and can assign partitions to members of the group. The code for that can be found here: https://github.com/NthMetal/gbraidfinder/blob/master/gbr/src/kafka.service.ts
To do this we start by assigning the partitions for the topic with the highest ranks because those have the most restrictions and the least amount of possible accounts they can be assigned to. Each partition is assigned to the account with the least heuristic value.
This heuristic value is calculated for an account by taking all of the currently assigned topics and multiplying the number of partitions in that topic by the current offset (which represents the amount of raids that topic gets).
The diagram below shows an example assignment withought weights.
Example weighted assignment with 10 accounts and 13 partitions:
Weights:
{ l80: '2990', l30: '319', l150: '25265', l120: '100499', l170: '3247', l20: '2', l130: '15595', l151: '31999', l200: '24856', l101: '58710', l40: '2384' }
Assignment (with weighted values shown):
{ 'gbr-217-00a9ea52-3ebf-472d-9b00-8cc6dab76ffa-70d3db3e-a993-4ec1-9db7-2020bbb40d3a': { // Heuristic Value (total): 335,585 l200: [ 0, 2, 4, 6, 8, 10, 12 ], // Weighted Value: 7 * 24856 = 173992 l120: [ 11 ], // Weighted Value: 1 * 100499 = 100499 l101: [ 9 ], // Weighted Value: 1 * 58710 = 58710 l40: [ 12 ] // Weighted Value: 1 * 2384 = 2384 }, 'gbr-206-58071966-ee27-4749-bb90-b82f0c597d11-331f2cd4-713f-4fef-a9c7-be5c039a89f4': { // Heuristic Value (total): 335,326 l200: [ 1, 3, 5, 7, 9, 11 ], // Weighted Value: 6 * 24856 = 149136 l130: [ 10 ], // Weighted Value: 1 * 15595 = 15595 l120: [ 7 ], // Weighted Value: 1 * 100499 = 100499 l101: [ 5 ], // Weighted Value: 1 * 58710 = 58710 l80: [ 5, 9 ], // Weighted Value: 2 * 2990 = 5980 l40: [ 1, 7 ], // Weighted Value: 2 * 2384 = 4768 l30: [ 5, 8 ] // Weighted Value: 2 * 319 = 638 }, 'gbr-187-be921fd0-860d-4068-b15a-7eaed3583358-8085ae9e-a35f-4219-a0de-e1a4580caf07': { // Heuristic Value (total): 335,375 l170: [ 0, 4, 8, 12 ], // Weighted Value: 4 * 3247 = 12988 l151: [ 4, 9 ], // Weighted Value: 2 * 31999 = 63998 l150: [ 1, 4, 9 ], // Weighted Value: 3 * 25265 = 75795 l130: [ 12 ], // Weighted Value: 1 * 15595 = 15595 l120: [ 9 ], // Weighted Value: 1 * 100499 = 100499 l101: [ 7 ], // Weighted Value: 1 * 58710 = 58710 l40: [ 0, 2, 8 ], // Weighted Value: 3 * 2384 = 7152 l30: [ 6, 9 ] // Weighted Value: 2 * 319 = 638 }, 'gbr-179-797843ac-4b1b-4584-9e0d-91a701c13eda-ec720c1d-1da6-4a3c-876b-472b47b74960': { // Heuristic Value (total): 374,187 l170: [ 1, 5, 9 ], // Weighted Value: 3 * 3247 = 9741 l151: [ 1, 6, 11 ], // Weighted Value: 3 * 31999 = 95997 l150: [ 5, 10 ], // Weighted Value: 2 * 25265 = 50530 l120: [ 5 ], // Weighted Value: 1 * 100499 = 100499 l101: [ 2, 12 ] // Weighted Value: 2 * 58710 = 117420 }, 'gbr-178-042f43ec-ec27-4839-95a4-1872d93c6997-c5d6474b-f095-4599-b100-0a62516fda54': { // Heuristic Value (total): 335,514 l170: [ 2, 6, 10 ], // Weighted Value: 3 * 3247 = 9741 l151: [ 2, 7, 12 ], // Weighted Value: 3 * 31999 = 95997 l150: [ 6, 11 ], // Weighted Value: 2 * 25265 = 50530 l120: [ 6 ], // Weighted Value: 1 * 100499 = 100499 l101: [ 3 ], // Weighted Value: 1 * 58710 = 58710 l80: [ 0, 2, 4, 8, 12 ], // Weighted Value: 5 * 2990 = 14950 l40: [ 5, 11 ], // Weighted Value: 2 * 2384 = 4768 l30: [ 12 ] // Weighted Value: 1 * 319 = 319 }, 'gbr-175-68c6d009-8996-40df-a64c-41492066cfe9-20e476c9-facd-41ec-9dbd-7a4f7a9657f4': { // Heuristic Value (total): 335,405 l170: [ 3, 7, 11 ], // Weighted Value: 3 * 3247 = 9741 l151: [ 3, 8 ], // Weighted Value: 2 * 31999 = 63998 l150: [ 0, 3, 8 ], // Weighted Value: 3 * 25265 = 75795 l130: [ 11 ], // Weighted Value: 1 * 15595 = 15595 l120: [ 8 ], // Weighted Value: 1 * 100499 = 100499 l101: [ 6 ], // Weighted Value: 1 * 58710 = 58710 l80: [ 6, 10 ], // Weighted Value: 2 * 2990 = 5980 l40: [ 3, 9 ], // Weighted Value: 2 * 2384 = 4768 l30: [ 10 ] // Weighted Value: 1 * 319 = 319 }, 'gbr-161-7806a832-01e7-4a89-a3ce-d8a649bc1b5f-c30eea20-ed03-4443-8a7a-2d1bee7d5a5a': { // Heuristic Value (total): 335,325 l151: [ 0, 5, 10 ], // Weighted Value: 3 * 31999 = 95997 l150: [ 2, 7, 12 ], // Weighted Value: 3 * 25265 = 75795 l120: [ 10 ], // Weighted Value: 1 * 100499 = 100499 l101: [ 8 ], // Weighted Value: 1 * 58710 = 58710 l40: [ 6 ], // Weighted Value: 1 * 2384 = 2384 l30: [ 0, 1, 2, 3, 4, 7 ], // Weighted Value: 6 * 319 = 1914 l20: [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 ] // Weighted Value: 13 * 2 = 26 }, 'gbr-146-0a297ca5-ddba-430c-b0c7-3d31e113c4b1-b7346f27-b403-49c5-95b0-13c8f79acf77': { // Heuristic Value (total): 373,869 l130: [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ], // Weighted Value: 10 * 15595 = 155950 l120: [ 4 ], // Weighted Value: 1 * 100499 = 100499 l101: [ 1, 11 ] // Weighted Value: 2 * 58710 = 117420 }, 'gbr-126-7ec4b018-668d-4f14-b997-2f4da680c057-4a1af67a-ea61-4f06-a745-f7db6744ba0b': { // Heuristic Value (total): 360,207 l120: [ 0, 2, 12 ], // Weighted Value: 3 * 100499 = 301497 l101: [ 10 ] // Weighted Value: 1 * 58710 = 58710 }, 'gbr-120-a6ac587c-a37d-4a92-ae7b-0e8739fca0bb-a488d853-4f38-4aed-b41b-fd41a31c3acc': { // Heuristic Value (total): 335,465 l120: [ 1, 3 ], // Weighted Value: 2 * 100499 = 200998 l101: [ 0, 4 ], // Weighted Value: 2 * 58710 = 117420 l80: [ 1, 3, 7, 11 ], // Weighted Value: 4 * 2990 = 11960 l40: [ 4, 10 ], // Weighted Value: 2 * 2384 = 4768 l30: [ 11 ] // Weighted Value: 1 * 319 = 319 } }
- Raid is tweeted and picked up by gbsource either through twitter or another source
- gbsource determines the raid quest and level requirement to join
- gbsource sends the raid message to the cooresponding level topic (the level prefixed with an "l" ex. l120)
- gbdist is subscribed to all level topics and picks up the raid and sends it to users that are subscribed
- at the same time a gbr that is assigned to the partition the message was sent in picks it up and gets the update (hp and other info)
- the gbr that got the updated info posts that message to the "update" topic
- gbdist is subscribed to the "update" topic and sends the updated info to users that got the original raid
Account Access through chrome debug port preview: