Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add policy definitions and attachments to request context, change pol… #141

Merged
merged 2 commits into from
Jun 22, 2020
Merged

Add policy definitions and attachments to request context, change pol… #141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5d3dc43
Add policy definitions and attachments to request context, change pol…
tomeresk Jun 18, 2020
2e656a9
PR comments
tomeresk Jun 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 32 additions & 7 deletions services/package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion services/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"version": "1.0.0",
"scripts": {
"test": "jest --config jest.config.js",
"test:debug": "node --inspect-brk node_modules/.bin/jest --runInBand --config jest.config.js",
"test:e2e": "jest --config jest.config.e2e.js",
"test:full": "jest --config jest.config.full.js",
"build": "tsc",
Expand Down Expand Up @@ -43,6 +44,7 @@
"@types/jest": "^25.1.3",
"@types/node": "^13.7.4",
"@types/pino": "^5.15.5",
"@types/xml2js": "^0.4.5",
"apollo-server-testing": "^2.10.1",
"docker-compose": "^0.23.3",
"graphql-request": "^1.8.2",
Expand All @@ -53,6 +55,7 @@
"nock": "^12.0.1",
"ts-jest": "^25.2.1",
"ts-node-dev": "^1.0.0-pre.44",
"typescript": "^3.7.5"
"typescript": "^3.7.5",
"xml2js": "^0.4.23"
}
}
7 changes: 1 addition & 6 deletions services/src/gateway.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,16 @@ import {
ResourceRepository,
CompositeResourceRepository,
} from './modules/resource-repository';
import {PolicyExecutor} from './modules/directives/policy/policy-executor';

async function run() {
logger.info('Stitch gateway booting up...');

const resourceRepository = getResourceRepository();

const {server, dispose} = createStitchGateway({
resourceGroups: pollForUpdates(resourceRepository, config.resourceUpdateInterval),
resourceGroups: pollForUpdates(getResourceRepository(), config.resourceUpdateInterval),
tracing: config.enableGraphQLTracing,
playground: config.enableGraphQLPlayground,
introspection: config.enableGraphQLIntrospection,
});
await resourceRepository.initializePolicyAttachments();
PolicyExecutor.repo = resourceRepository;

const app = fastify();
app.register(fastifyMetrics, {endpoint: '/metrics'});
Expand Down
2 changes: 1 addition & 1 deletion services/src/modules/directives/policy/opa.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ export async function evaluate(ctx: PolicyExecutionContext): Promise<PolicyExecu

async function getWasmPolicy(ctx: PolicyExecutionContext): Promise<any> {
const filename = getCompiledFilename({namespace: ctx.namespace, name: ctx.name});
const wasm = ctx.repo.getPolicyAttachment(filename);
const wasm = ctx.policyAttachments[filename];

const rego = new Rego();
return rego.load_policy(wasm);
Expand Down
9 changes: 5 additions & 4 deletions services/src/modules/directives/policy/policy-executor.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import {GraphQLResolveInfo} from 'graphql';
import {RequestContext} from '../../context';
import {Policy, GraphQLArguments} from './types';
import {ResourceRepository, Policy as PolicyDefinition, PolicyArgsObject} from '../../resource-repository';
import {Policy as PolicyDefinition, PolicyArgsObject, PolicyAttachments} from '../../resource-repository';
import {evaluate as evaluateOpa} from './opa';
import {injectParameters} from '../../paramInjection';

Expand All @@ -10,8 +10,8 @@ const typeEvaluators = {
};

export class PolicyExecutor {
static repo: ResourceRepository;
private policyDefinitions: PolicyDefinition[];
private policyAttachments: PolicyAttachments;

constructor(
protected policies: Policy[],
Expand All @@ -21,7 +21,8 @@ export class PolicyExecutor {
protected info: GraphQLResolveInfo
) {
// TODO: add jwt data
this.policyDefinitions = PolicyExecutor.repo.getResourceGroup().policies;
this.policyDefinitions = context.policies;
this.policyAttachments = context.policyAttachments;
}

async validatePolicies() {
Expand All @@ -37,7 +38,7 @@ export class PolicyExecutor {
const evaluate = typeEvaluators[policyDefinition.type];
if (!evaluate) throw new Error(`Unsupported policy type ${policyDefinition.type}`);

const {done, allow} = await evaluate({...policy, args, repo: PolicyExecutor.repo});
const {done, allow} = await evaluate({...policy, args, policyAttachments: this.policyAttachments});
if (!done) throw new Error('in-line query evaluation not yet supported');

if (!allow) throw new Error(`Unauthorized by policy ${policy.name} in namespace ${policy.namespace}`);
Expand Down
4 changes: 2 additions & 2 deletions services/src/modules/directives/policy/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import {PolicyArgsObject, ResourceRepository} from '../../resource-repository/types';
import {PolicyArgsObject, PolicyAttachments} from '../../resource-repository/types';

export type Policy = {
namespace: string;
Expand All @@ -10,7 +10,7 @@ export type Policy = {
export type PolicyExecutionContext = {
namespace: string;
name: string;
repo: ResourceRepository;
policyAttachments: PolicyAttachments;
jwt?: JwtInput;
args?: PolicyArgsObject;
queries?: QueriesResults;
Expand Down
6 changes: 5 additions & 1 deletion services/src/modules/graphqlService.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import {Observable, Subscription} from 'rxjs';
import {shareReplay, map, take, tap, catchError, skip} from 'rxjs/operators';

import {directiveMap} from './directives';
import {ResourceGroup} from './resource-repository';
import {ResourceGroup, Policy, PolicyAttachments} from './resource-repository';
import {buildSchemaFromFederatedTypeDefs} from './buildFederatedSchema';
import * as baseSchema from './baseSchema';
import {ActiveDirectoryAuth} from './auth/activeDirectoryAuth';
Expand Down Expand Up @@ -86,6 +86,8 @@ export function createSchemaConfig(rg: ResourceGroup): GraphQLServiceConfig {
schema,
executor(requestContext) {
requestContext.context.authenticationConfig = authenticationConfig;
requestContext.context.policies = rg.policies;
requestContext.context.policyAttachments = rg.policyAttachments;

return execute({
document: requestContext.document,
Expand All @@ -106,5 +108,7 @@ const defaultSchema = {
declare module './context' {
interface RequestContext {
authenticationConfig: AuthenticationConfig;
policies: Policy[];
policyAttachments: PolicyAttachments;
}
}
21 changes: 1 addition & 20 deletions services/src/modules/resource-repository/composite.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import {ResourceRepository, FetchLatestResult, ResourceGroup} from './types';
import {ResourceRepository, FetchLatestResult} from './types';
import {applyResourceGroupUpdates} from './updates';

export class CompositeResourceRepository implements ResourceRepository {
Expand All @@ -13,30 +13,11 @@ export class CompositeResourceRepository implements ResourceRepository {
}));
}

getResourceGroup(): ResourceGroup {
const rgs = this.repositories.map(r => r.getResourceGroup());

return rgs.reduce((rg1, rg2) => applyResourceGroupUpdates(rg1, rg2));
}

async update(): Promise<void> {
throw new Error('Multiplexed resource repository cannot handle updates');
}

async writePolicyAttachment(): Promise<void> {
throw new Error('Multiplexed resource repository cannot handle updates');
}

public getPolicyAttachment(filename: string): Buffer {
for (const repo of this.repositories) {
const policyAttachment = repo.getPolicyAttachment(filename);
if (policyAttachment) return policyAttachment;
}

throw new Error(`Policy attachment with the filename ${filename} was not found`);
}

public async initializePolicyAttachments() {
await Promise.all(this.repositories.map(repo => repo.initializePolicyAttachments()));
}
}
37 changes: 5 additions & 32 deletions services/src/modules/resource-repository/fs.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,12 @@ import * as envVar from 'env-var';
import {promises as fs} from 'fs';
import * as path from 'path';
import pLimit from 'p-limit';
import * as config from '../config';
import logger from '../logger';
import {FetchLatestResult} from './types';
import {FetchLatestResult, PolicyAttachments} from './types';

export class FileSystemResourceRepository implements ResourceRepository {
protected current?: {mtime: number; rg: ResourceGroup};
protected policyAttachmentsDirInitialized = false;
protected policyAttachments: {[filename: string]: Buffer} = {};
protected policyAttachments: PolicyAttachments = {};
protected policyAttachmentsRefreshedAt?: Date;

constructor(protected pathToFile: string, protected policyAttachmentsFolderPath: string) {}
Expand All @@ -26,11 +24,10 @@ export class FileSystemResourceRepository implements ResourceRepository {
const rg = JSON.parse(contents) as ResourceGroup;
this.current = {mtime: stats.mtimeMs, rg};

return {isNew: true, resourceGroup: rg};
}
await this.refreshPolicyAttachments();
rg.policyAttachments = {...this.policyAttachments};

getResourceGroup(): ResourceGroup {
return this.current!.rg;
return {isNew: true, resourceGroup: rg};
}

async update(rg: ResourceGroup): Promise<void> {
Expand All @@ -44,30 +41,6 @@ export class FileSystemResourceRepository implements ResourceRepository {
await fs.writeFile(filePath, content);
}

public getPolicyAttachment(filename: string): Buffer {
return this.policyAttachments[filename];
}

public async initializePolicyAttachments() {
try {
await this.refreshPolicyAttachments();
} catch (err) {
logger.fatal({err}, 'Failed fetching fs policy attachments on startup');
throw err;
}

setInterval(async () => {
try {
await this.refreshPolicyAttachments();
} catch (err) {
logger.error(
{err},
`Failed refreshing fs policy attachments, last successful refresh was at ${this.policyAttachmentsRefreshedAt}`
);
}
}, config.resourceUpdateInterval);
}

private async refreshPolicyAttachments() {
const newRefreshedAt = new Date();

Expand Down
37 changes: 5 additions & 32 deletions services/src/modules/resource-repository/s3.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
import * as AWS from 'aws-sdk';
import * as envVar from 'env-var';
import pLimit from 'p-limit';
import * as config from '../config';
import logger from '../logger';
import {ResourceRepository, ResourceGroup, FetchLatestResult} from './types';
import {ResourceRepository, ResourceGroup, FetchLatestResult, PolicyAttachments} from './types';

interface S3ResourceRepositoryConfig {
s3: AWS.S3;
Expand All @@ -15,7 +13,7 @@ type FileDetails = {filename: string; updatedAt: Date};

export class S3ResourceRepository implements ResourceRepository {
protected current?: {etag?: string; rg: ResourceGroup};
protected policyAttachments: {[filename: string]: Buffer} = {};
protected policyAttachments: PolicyAttachments = {};
protected policyAttachmentsRefreshedAt?: Date;

constructor(protected config: S3ResourceRepositoryConfig) {}
Expand Down Expand Up @@ -48,16 +46,15 @@ export class S3ResourceRepository implements ResourceRepository {
const rg = JSON.parse(bodyRaw) as ResourceGroup;
this.current = {etag: response.ETag, rg};

await this.refreshPolicyAttachments();
rg.policyAttachments = {...this.policyAttachments};

return {
isNew: true,
resourceGroup: rg,
};
}

getResourceGroup(): ResourceGroup {
return this.current!.rg;
}

async update(rg: ResourceGroup): Promise<void> {
await this.config.s3
.putObject({
Expand All @@ -80,30 +77,6 @@ export class S3ResourceRepository implements ResourceRepository {
.promise();
}

public getPolicyAttachment(filename: string): Buffer {
return this.policyAttachments[filename];
}

public async initializePolicyAttachments() {
try {
await this.refreshPolicyAttachments();
} catch (err) {
logger.fatal({err}, 'Failed fetching s3 policy attachments on startup');
throw err;
}

setInterval(async () => {
try {
await this.refreshPolicyAttachments();
} catch (err) {
logger.error(
{err},
`Failed refreshing s3 policy attachments, last successful refresh was at ${this.policyAttachmentsRefreshedAt}`
);
}
}, config.resourceUpdateInterval);
}

private async refreshPolicyAttachments() {
const newRefreshedAt = new Date();

Expand Down
7 changes: 4 additions & 3 deletions services/src/modules/resource-repository/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ export interface ResourceGroup {
upstreams: Upstream[];
upstreamClientCredentials: UpstreamClientCredentials[];
policies: Policy[];
// policyAttachments are compiled from the Rego code in opa policies, they are not directly modified by users
policyAttachments?: PolicyAttachments;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leave a comment about this being a "runtime-only" property? Since the rest are assumed to always exist

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Come to think of it, it will be populated on the first call to fetchLatest same as the others, and will always exist (empty object if there are no attachments at all, same as the other fields here).
All of them are actually runtime-only properties representing what is stored in the underlying repository, with the only different between the attachments and the other properties being that the other properties are all stored in one file and the attachments are stored in other files.

However, when using this interface we don't really care that it is stored in other files than the other properties, so I'm actually leaning towards removing the ? here and treating it like any other property.

One key difference is that the contents of this property are calculated based on the policies property and not created/updated directly by a user, but it only happens once when we save the policy and then saved this way in the repository, so I would still say we should treat it like the other properties here (but I will add a comment to highlight this difference)

}

export interface Resource {
Expand Down Expand Up @@ -70,13 +72,12 @@ export interface FetchLatestResult {
resourceGroup: ResourceGroup;
}

export type PolicyAttachments = {[filename: string]: Buffer};

export interface ResourceRepository {
fetchLatest(): Promise<FetchLatestResult>;
getResourceGroup(): ResourceGroup;
update(rg: ResourceGroup): Promise<void>;
writePolicyAttachment(filename: string, content: Buffer): Promise<void>;
getPolicyAttachment(filename: string): Buffer;
initializePolicyAttachments(): Promise<void>;
}

enum AuthType {
Expand Down
Loading