Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use new hashing service #2896

Merged
Merged
Show file tree
Hide file tree
Changes from 69 commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
e437ebe
Use hashinService insteast validationModuleManager for new functions
Mihajlo-Pavlovic Jan 16, 2024
2c96044
Update sharding-table-service to use proximityScoringService
Mihajlo-Pavlovic Jan 16, 2024
98c2001
Update findNeighbourhood function calls to use proximityScoreFunction…
Mihajlo-Pavlovic Jan 16, 2024
52567ef
Update epoch-check-command
Mihajlo-Pavlovic Jan 16, 2024
bf7ed5a
Update calculateScore function in ServiceAgreementService to use Prox…
Mihajlo-Pavlovic Jan 16, 2024
62665ec
Update commands to use proximityScoreFunctionsPairId
Mihajlo-Pavlovic Jan 16, 2024
f9ba6d8
Update bid sugestion controller
Mihajlo-Pavlovic Jan 17, 2024
164ccd6
Update operation commands
Mihajlo-Pavlovic Jan 17, 2024
902838f
Update rpc-controllers to handle reciving proximityScoreFunctionsPairId
Mihajlo-Pavlovic Jan 17, 2024
2cb4d7b
Use getAgreementScoreFunctionId to get proximityScoreFunctionsPairId …
Mihajlo-Pavlovic Jan 17, 2024
0d4f0ef
Update bid-suggerstion-schema to include proximityScoreFunctionsPairId
Mihajlo-Pavlovic Jan 17, 2024
6d5747f
Add test to check publish, get, update handling of hashFunctionId & s…
Mihajlo-Pavlovic Jan 18, 2024
f02a2a5
Add bid suggestion tests
Mihajlo-Pavlovic Jan 18, 2024
85b1d4a
Fix updated code
Mihajlo-Pavlovic Jan 18, 2024
98781de
Add scenario
Mihajlo-Pavlovic Jan 18, 2024
3bb9abd
Fixes
Mihajlo-Pavlovic Jan 19, 2024
2ba2fa8
Merge branch 'feature/use-new-hashing-service' into test/hash-and-sco…
Mihajlo-Pavlovic Jan 19, 2024
7c9b6ea
Update test scenario
Mihajlo-Pavlovic Jan 19, 2024
641c2f5
Fix scenario typo
Mihajlo-Pavlovic Jan 22, 2024
c9d8c7d
Update dkg.js version
Mihajlo-Pavlovic Jan 22, 2024
b8b4e0b
Fix getBidSuggestion tests
Mihajlo-Pavlovic Jan 22, 2024
0159220
Use big number comparison
Mihajlo-Pavlovic Jan 24, 2024
4dc25dd
Merge remote-tracking branch 'origin/feature/final-proximity-scoring-…
Mihajlo-Pavlovic Jan 24, 2024
e9b2480
Merge remote-tracking branch 'origin/feature/final-proximity-scoring-…
Mihajlo-Pavlovic Jan 24, 2024
b89590d
Find asset neighbourhood edge
Mihajlo-Pavlovic Jan 24, 2024
03867d0
Merge branch 'test/hash-and-scoredistnace-function-id' into feature/u…
Mihajlo-Pavlovic Jan 25, 2024
c14ecf8
Added new migration
djordjekovac Jan 25, 2024
3fdbeb5
Implement findEdges method
Mihajlo-Pavlovic Jan 25, 2024
5d53b5f
Add index to neighborhood peers
Mihajlo-Pavlovic Jan 25, 2024
6e56349
Fix index addition
Mihajlo-Pavlovic Jan 25, 2024
bdaaeca
Merge pull request #2897 from OriginTrail/v6/feature/remove-service-a…
djordjekovac Jan 25, 2024
ccfbb58
Store sha256 as blob in sql
Mihajlo-Pavlovic Jan 29, 2024
8b05250
Add liearSum params get
Mihajlo-Pavlovic Jan 29, 2024
55af92e
Get rank fix
Mihajlo-Pavlovic Jan 29, 2024
efb3d65
Added proximity score functions pair id where we need to choose commi…
djordjekovac Jan 29, 2024
91bfb2c
Update caluclateRank in update command
Mihajlo-Pavlovic Jan 29, 2024
036afc2
submitUpdateCommitCommand pass correct data
Mihajlo-Pavlovic Jan 29, 2024
8406d38
Update submitUpdateCommitCommand logs
Mihajlo-Pavlovic Jan 29, 2024
40280c3
Reverted changes
djordjekovac Jan 29, 2024
fdbab89
moved calculate rank and find neighbourhood edges to service classes
djordjekovac Jan 29, 2024
c7730f3
Remove sha256Blob from select allPears
Mihajlo-Pavlovic Jan 29, 2024
c296795
Resolved bug with neighbourhood edges in epoch check command
djordjekovac Jan 29, 2024
bef165d
Resolved bug with sending submit commit
djordjekovac Jan 29, 2024
8b2d1b1
Call serviceAgreementService.calculateRank in handle update request
Mihajlo-Pavlovic Jan 29, 2024
e02a1f8
Updated caching of params
djordjekovac Jan 29, 2024
3f22072
Merge branch 'feature/use-new-hashing-service' of https://github.com/…
djordjekovac Jan 29, 2024
9548302
Updated contract call caching, updated sharding table fetching
djordjekovac Jan 29, 2024
6b2243a
Updated constants
djordjekovac Jan 29, 2024
79bfedf
Fix pulling of sharding table;
NZT48 Jan 29, 2024
438ce4d
Add commit manager v2 contracts
NZT48 Jan 30, 2024
be68bc4
Add ToDo comments
NZT48 Jan 30, 2024
93b5c8a
Update src/commands/protocols/common/submit-commit-command.js
NZT48 Jan 30, 2024
1d54d10
Updated pull sharding table
djordjekovac Jan 30, 2024
cb74688
Updated contracts in web3service
djordjekovac Jan 30, 2024
6f30c24
Updated key calculation
djordjekovac Jan 30, 2024
2e9deea
Updated submit commit method signature
djordjekovac Jan 30, 2024
b973439
Updated submit commit function call
djordjekovac Jan 30, 2024
beb7ba3
Fix _executeContractFunction gasLimit error handling
Mihajlo-Pavlovic Jan 30, 2024
0dc8723
Added logs for sharding table
djordjekovac Jan 30, 2024
bf2b3e7
Updated max distance calculation for hash ring position
djordjekovac Jan 30, 2024
27649a3
Fixed submit commit contract call
djordjekovac Jan 30, 2024
dffdd03
Fix submitUpdateCommit
Mihajlo-Pavlovic Jan 30, 2024
db6ba3f
Check if edges exist befor looking for index
NZT48 Jan 30, 2024
e775305
Merge branch 'feature/use-new-hashing-service' of https://github.com/…
NZT48 Jan 30, 2024
dec0b0c
Removed logs
djordjekovac Jan 30, 2024
4a14b9c
Set dkg-evm-module dependecy to github repo
NZT48 Jan 30, 2024
958c40f
Remove async; from shard-repository
Mihajlo-Pavlovic Jan 30, 2024
2f3043a
Update src/controllers/http-api/v0/bid-suggestion-http-api-controller…
NZT48 Jan 30, 2024
aacf13e
Set dkg-evm-module dependecy to github repo
NZT48 Jan 30, 2024
4a2547d
Fixed formula for LinearSum score calculation to be consistent with t…
0xbraindevd Jan 30, 2024
2886775
Reverted back dependency for dkg-evm-module
0xbraindevd Jan 30, 2024
f69260e
Merge pull request #2898 from OriginTrail/fix/linear-sum-formula
NZT48 Jan 30, 2024
4209640
Added casting for LinearSum parameters
0xbraindevd Jan 30, 2024
a4755a6
Update package-lock with latest dkg-evm-module commit
NZT48 Jan 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions ot-node.js
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,12 @@ class OTNode {

await this.initializeModules();

await MigrationExecutor.executeRemoveServiceAgreementsForChiadoDevnetMigration(
this.container,
this.logger,
this.config,
);

await this.createProfiles();

await this.initializeCommandExecutor();
Expand Down
47 changes: 23 additions & 24 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
"chai": "^4.3.6",
"d3": "^7.8.5",
"d3-node": "^3.0.0",
"dkg.js": "^6.0.2",
"dkg.js": "^6.1.2",
"eslint": "^8.23.0",
"eslint-config-airbnb": "^19.0.4",
"eslint-config-prettier": "^8.5.0",
Expand Down Expand Up @@ -80,7 +80,7 @@
"axios": "^1.6.0",
"cors": "^2.8.5",
"deep-extend": "^0.6.0",
"dkg-evm-module": "^4.1.0",
"dkg-evm-module": "github:OriginTrail/dkg-evm-module#release/delegations",
"dotenv": "^16.0.1",
"ethers": "^5.7.2",
"express": "^4.18.1",
Expand Down
75 changes: 40 additions & 35 deletions src/commands/protocols/common/epoch-check-command.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ class EpochCheckCommand extends Command {
this.blockchainModuleManager = ctx.blockchainModuleManager;
this.serviceAgreementService = ctx.serviceAgreementService;
this.fileService = ctx.fileService;
this.proximityScoringService = ctx.proximityScoringService;
this.hashingService = ctx.hashingService;

this.errorType = ERROR_TYPE.COMMIT_PROOF.EPOCH_CHECK_ERROR;
}
Expand Down Expand Up @@ -124,12 +126,39 @@ class EpochCheckCommand extends Command {
for (const serviceAgreement of eligibleAgreementForSubmitCommit) {
if (scheduleSubmitCommitCommands.length >= maxTransactions) break;

const neighbourhood = await this.shardingTableService.findNeighbourhood(
blockchain,
serviceAgreement.keyword,
r2,
serviceAgreement.hashFunctionId,
serviceAgreement.scoreFunctionId,
true,
);

let neighbourhoodEdges = null;
if (serviceAgreement.scoreFunctionId === 2) {
neighbourhoodEdges = await this.shardingTableService.getNeighboorhoodEdgeNodes(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep American version neighborhood consistently everywhere

Suggested change
neighbourhoodEdges = await this.shardingTableService.getNeighboorhoodEdgeNodes(
neighborhoodEdges = await this.shardingTableService.getNeighborhoodEdgeNodes(

neighbourhood,
blockchain,
serviceAgreement.hashFunctionId,
serviceAgreement.scoreFunctionId,
serviceAgreement.keyword,
);
}

if (!neighbourhoodEdges && serviceAgreement.scoreFunctionId === 2) {
throw Error('Unable to find neighbourhood edges for asset');
}

try {
const rank = await this.calculateRank(
const rank = await this.serviceAgreementService.calculateRank(
blockchain,
serviceAgreement.keyword,
serviceAgreement.hashFunctionId,
serviceAgreement.scoreFunctionId,
r2,
neighbourhood,
neighbourhoodEdges,
);

updateServiceAgreementsLastCommitEpoch.push(
Expand Down Expand Up @@ -164,9 +193,13 @@ class EpochCheckCommand extends Command {
serviceAgreement.agreementId
}. Scheduling submitCommitCommand.`,
);

const closestNode = neighbourhood[0];
scheduleSubmitCommitCommands.push(
this.scheduleSubmitCommitCommand(serviceAgreement),
this.scheduleSubmitCommitCommand(
serviceAgreement,
neighbourhoodEdges,
closestNode,
),
);
} catch (error) {
this.logger.warn(
Expand Down Expand Up @@ -239,37 +272,6 @@ class EpochCheckCommand extends Command {
]);
}

async calculateRank(blockchain, keyword, hashFunctionId, r2) {
const neighbourhood = await this.shardingTableService.findNeighbourhood(
blockchain,
keyword,
r2,
hashFunctionId,
true,
);

const peerId = this.networkModuleManager.getPeerId().toB58String();
if (!neighbourhood.some((node) => node.peerId === peerId)) {
return;
}

const scores = await Promise.all(
neighbourhood.map(async (node) => ({
score: await this.serviceAgreementService.calculateScore(
node.peerId,
blockchain,
keyword,
hashFunctionId,
),
peerId: node.peerId,
})),
);

scores.sort((a, b) => b.score - a.score);

return scores.findIndex((node) => node.peerId === peerId);
}

async isEligibleForRewards(blockchain, agreementId, epoch, stateIndex, r0) {
const identityId = await this.blockchainModuleManager.getIdentityId(blockchain);
const commits = await this.blockchainModuleManager.getTopCommitSubmissions(
Expand All @@ -288,7 +290,7 @@ class EpochCheckCommand extends Command {
return false;
}

async scheduleSubmitCommitCommand(agreement) {
async scheduleSubmitCommitCommand(agreement, neighbourhoodEdges, closestNode) {
const commandData = {
operationId: this.operationIdService.generateId(),
blockchain: agreement.blockchainId,
Expand All @@ -299,6 +301,9 @@ class EpochCheckCommand extends Command {
epoch: agreement.currentEpoch,
agreementId: agreement.agreementId,
stateIndex: agreement.stateIndex,
closestNode: closestNode.index,
leftNeighborhoodEdge: neighbourhoodEdges?.leftEdge.index,
rightNeighborhoodEdge: neighbourhoodEdges?.rightEdge.index,
};

await this.commandExecutor.add({
Expand Down
18 changes: 16 additions & 2 deletions src/commands/protocols/common/find-nodes-command.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,21 @@ class FindNodesCommand extends Command {
networkProtocols,
hashFunctionId,
minAckResponses,
proximityScoreFunctionsPairId,
} = command.data;

this.errorType = errorType;
this.logger.debug(`Searching for closest node(s) for keyword ${keyword}`);

// TODO: protocol selection
const closestNodes = [];
const foundNodes = await this.findNodes(blockchain, keyword, operationId, hashFunctionId);
const foundNodes = await this.findNodes(
blockchain,
keyword,
operationId,
hashFunctionId,
proximityScoreFunctionsPairId,
);
for (const node of foundNodes) {
if (node.id !== this.networkModuleManager.getPeerId().toB58String()) {
closestNodes.push({ id: node.id, protocol: networkProtocols[0] });
Expand Down Expand Up @@ -66,7 +73,13 @@ class FindNodesCommand extends Command {
);
}

async findNodes(blockchainId, keyword, operationId, hashFunctionId) {
async findNodes(
blockchainId,
keyword,
operationId,
hashFunctionId,
proximityScoreFunctionsPairId,
) {
await this.operationIdService.updateOperationIdStatus(
operationId,
blockchainId,
Expand All @@ -78,6 +91,7 @@ class FindNodesCommand extends Command {
keyword,
r2,
hashFunctionId,
proximityScoreFunctionsPairId,
true,
);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,19 @@ class HandleProtocolMessageCommand extends Command {
throw Error('prepareMessage not implemented');
}

async validateNeighborhood(blockchain, keyword, hashFunctionId, ual) {
async validateNeighborhood(
blockchain,
keyword,
hashFunctionId,
proximityScoreFunctionsPairId,
ual,
) {
const closestNodes = await this.shardingTableService.findNeighbourhood(
blockchain,
keyword,
await this.blockchainModuleManager.getR2(blockchain),
hashFunctionId,
proximityScoreFunctionsPairId,
true,
);
const peerId = this.networkModuleManager.getPeerId().toB58String();
Expand Down Expand Up @@ -179,11 +186,20 @@ class HandleProtocolMessageCommand extends Command {
tokenId,
keyword,
hashFunctionId,
proximityScoreFunctionsPairId,
) {
const ual = this.ualService.deriveUAL(blockchain, contract, tokenId);

this.logger.trace(`Validating neighborhood for ual: ${ual}`);
if (!(await this.validateNeighborhood(blockchain, keyword, hashFunctionId, ual))) {
if (
!(await this.validateNeighborhood(
blockchain,
keyword,
hashFunctionId,
proximityScoreFunctionsPairId,
ual,
))
) {
return {
messageType: NETWORK_MESSAGE_TYPES.RESPONSES.NACK,
messageData: { errorMessage: 'Invalid neighbourhood' },
Expand Down
18 changes: 17 additions & 1 deletion src/commands/protocols/common/network-protocol-command.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,34 @@ class NetworkProtocolCommand extends Command {
constructor(ctx) {
super(ctx);
this.commandExecutor = ctx.commandExecutor;
this.blockchainModuleManager = ctx.blockchainModuleManager;
this.serviceAgreementService = ctx.serviceAgreementService;
}

/**
* Executes command and produces one or more events
* @param command
*/
async execute(command) {
const { blockchain } = command.data;
const { blockchain, contract, tokenId, hashFunctionId } = command.data;

const keywords = await this.getKeywords(command);
const batchSize = await this.getBatchSize(blockchain);
const minAckResponses = await this.getMinAckResponses(blockchain);

const serviceAgreementId = await this.serviceAgreementService.generateId(
blockchain,
contract,
tokenId,
keywords[0],
hashFunctionId,
);
const proximityScoreFunctionsPairId =
await this.blockchainModuleManager.getAgreementScoreFunctionId(
blockchain,
serviceAgreementId,
);

const commandSequence = [
'findNodesCommand',
`${this.operationService.getOperationName()}ScheduleMessagesCommand`,
Expand All @@ -34,6 +49,7 @@ class NetworkProtocolCommand extends Command {
minAckResponses,
errorType: this.errorType,
networkProtocols: this.operationService.getNetworkProtocols(),
proximityScoreFunctionsPairId,
},
transactional: false,
}),
Expand Down
Loading
Loading