Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changed ID to be case sensitive #227

Merged
merged 2 commits into from
Feb 23, 2022
+90 −89
Merged

changed ID to be case sensitive #227

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b19aeba
changed ID to be case sensitive
kristinapathak Feb 23, 2022
233b238
fix tests, prep for release
kristinapathak Feb 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [v0.7.0]
- Updated spec file and rpkg version macro to be able to choose when the 'v' is included in the version. [#224](https://github.com/xmidt-org/argus/pull/224)
- Make ID case sensitive. [#227](https://github.com/xmidt-org/argus/pull/227)

## [v0.6.0]
- Split Chrysom client into BasicClient and ListenerClient. [#206](https://github.com/xmidt-org/argus/pull/206)
Expand Down Expand Up @@ -185,7 +188,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
## [v0.1.0] Tue May 07 2020 Jack Murdock - 0.1.0
- initial creation

[Unreleased]: https://github.com/xmidt-org/argus/compare/v0.6.0...HEAD
[Unreleased]: https://github.com/xmidt-org/argus/compare/v0.7.0...HEAD
[v0.7.0]: https://github.com/xmidt-org/argus/compare/v0.6.0...v0.7.0
[v0.6.0]: https://github.com/xmidt-org/argus/compare/v0.5.2...v0.6.0
[v0.5.2]: https://github.com/xmidt-org/argus/compare/v0.5.1...v0.5.2
[v0.5.1]: https://github.com/xmidt-org/argus/compare/v0.5.0...v0.5.1
Expand Down
119 changes: 81 additions & 38 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@
[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=xmidt-org_argus&metric=alert_status)](https://sonarcloud.io/dashboard?id=xmidt-org_argus)

## Summary
The [XMiDT](https://xmidt.io/) server for storing webhooks to be used by caduceus. This service is used to replace SNS.
Refer the [overview docs](https://xmidt.io/docs/introduction/overview/) for more information on how argus fits into the overall picture.
The [XMiDT](https://xmidt.io/) server for storing webhooks to be used by
caduceus. This service is used to replace SNS. Refer the [overview docs](https://xmidt.io/docs/introduction/overview/)
for more information on how Argus fits into the overall picture.

## Table of Contents

Expand All @@ -27,36 +28,52 @@ This project and everyone participating in it are governed by the [XMiDT Code Of
By participating, you agree to this Code.

## Details
argus has one function: interact with a database whether it is internal or external.
To enable this, argus has two endpoints: 1) individual items, and 2) buckets containing items.
Argus has one function: interact with a database whether it is internal or
external. To enable this, Argus has two endpoints: 1) individual items, and 2)
buckets containing items.

### Create Individual Item - `store/{bucket}/{id}` endpoint
This endpoint allows for clients to `PUT` an object into argus. The placeholder variables in the path must contain:
This endpoint allows for clients to `PUT` an object into Argus. The placeholder variables in the path must contain:

* _bucket_ - The name used to indicate the resource type of which the stored data represents. A plural form of a noun word should be used for stylistic reasons.
By default, the following rules will be enforced:
* _bucket_ - The name used to indicate the resource type of which the stored
data represents. A plural form of a noun word should be used for stylistic
reasons. By default, the following rules will be enforced:
- Bucket names must be between 3 and 63 characters long.
- Bucket names can consist only of lowercase letters, numbers and hyphens (-).
- Bucket names must begin and end with a letter or number.

If you'd like to define your own bucket validation format, check out the `userInputValidation.bucketFormatRegex` configuration option.
* _id_ - The unique ID within the name space of the containing bucket. It is recommended this value is the resulting value of a SHA256 calculation, using the unique attributes of the object being represented (e.g. `SHA256(<common_name>)`). This will be used by argus to determine uniqueness of objects being stored or updated. argus will not accept any values for this attribute that is not a 64 character hex string.

**Note:** While Argus is case insensitive for its IDs, all responses will always return their lowercase form regardless of the casing of your input ID.


* _ID_ - The unique ID within the name space of the containing bucket. It is
recommended this value is the resulting value of a SHA256 calculation, using
the unique attributes of the object being represented (e.g.
`SHA256(<common_name>)`). This will be used by Argus to determine uniqueness
of objects being stored or updated. Argus will not accept any values for this
attribute that is not a 64 character hex string containing only 0-9 and a-f.

The body must be in JSON format with the following attributes:

* _id_ - Required. See above.
* _data_ - Required. RAW JSON to be stored. Opaque to argus.
* _ttl_ - Optional. Specified in units of seconds. Defaults to the value of the server configuration option `itemMaxTTL`. If a configuration value is not specified, the value would be a day (~ 24*60^2 seconds).
)

An optional header `X-Midt-Owner` can be sent to associate the object with an owner. The value of this header will be bound to the new item, which would require the same value passed in a `X-Midt-Owner` header for subsequent reads or modifications. This in effect creates a secret attribute bound to the life of newly created items. When provided, Argus validates the length of the owner string to be in the range `[10,60]`. If you'd like to define your own validation format, check out the `userInputValidation.ownerFormatRegex` configuration option.
* _ID_ - Required. Must match the ID provided in the URL.
* _data_ - Required. RAW JSON to be stored. Opaque to Argus.
* _ttl_ - Optional. Specified in units of seconds. Defaults to the value of
the server configuration option `itemMaxTTL`. If a configuration value is not
specified, the value would be a day (~ 24*60^2 seconds). )

An optional header `X-Midt-Owner` can be sent to associate the object with an
owner. The value of this header will be bound to the new item, which would
require the same value passed in a `X-Midt-Owner` header for subsequent reads or
modifications. This in effect creates a secret attribute bound to the life of
newly created items. When provided, Argus validates the length of the owner
string to be in the range `[10,60]`. If you'd like to define your own validation
format, check out the `userInputValidation.ownerFormatRegex` configuration
option.

When the header is not provided, the owner of the item will be the empty string.

The exception to the above would be an authorized request. The authorization method is not specified and is up to the implementation to decide. Authorized requests shall be allowed to update all attributes except the `X-Midt-Owner` meta attribute.
The exception to the above would be an authorized request. The authorization
method is not specified and is up to the implementation to decide. Authorized
requests shall be allowed to update all attributes except the `X-Midt-Owner`
meta attribute.

An example PUT request
```
Expand All @@ -77,18 +94,32 @@ Example responses:
```
HTTP/1.1 201 Created
```
The above response would indicate a new object has been created (no existing object with the given ID was found).
The above response would indicate a new object has been created (no existing
object with the given ID was found).

```
HTTP/1.1 200 OK
```
The above response would indicate an existing object has been updated (existing object with the given ID was found). Note that a PUT operation on an existing record may also result in "403 Forbidden" error.
The above response would indicate an existing object has been updated (existing
object with the given ID was found). Note that a PUT operation on an existing
record may also result in "403 Forbidden" error.

**Note:** If a service using Argus must submit JSON data with duplicate fields,
please see [this](https://github.com/xmidt-org/argus/issues/60) issue for
details on expected behavior.

**Note:** If a service using Argus must submit JSON data with duplicate fields, please see [this](https://github.com/xmidt-org/argus/issues/60) issue for details on expected behavior.
### List - `store/{bucket}` endpoint
This endpoint allows for `GET` to retrieve all the items in the bucket organized by the id.

An example response will look like the below where "7e8c5f378b4addbaebc70897c4478cca06009e3e360208ebd073dbee4b3774e7" is the id of the only item in this collection. An optional header `X-Midt-Owner` can be sent with the request. If supplied, only items with secrets matching the supplied value will be returned in the list. If not supplied, all items created without an owner (owner value = "") will be returned. For authorized requests, if no owner header is provided, all items for the specified bucket will be returned.
This endpoint allows for `GET` to retrieve all the items in the bucket organized
by the id.

An example response will look like the below where
"7e8c5f378b4addbaebc70897c4478cca06009e3e360208ebd073dbee4b3774e7" is the id of
the only item in this collection. An optional header `X-Midt-Owner` can be sent
with the request. If supplied, only items with secrets matching the supplied
value will be returned in the list. If not supplied, all items created without
an owner (owner value = "") will be returned. For authorized requests, if no
owner header is provided, all items for the specified bucket will be returned.

An example response:
```json
Expand All @@ -109,9 +140,17 @@ An example response:
]
```


### Individual Item - `store/{bucket}/{id}` endpoint
This endpoint allows for `GET`, and `DELETE` REST methods to interact with any object that was created with the previous `PUT` request. An optional header `X-Midt-Owner` can be sent with the request. All requests are validated by comparing the secret stored with the requested record with the value sent in the `X-Midt-Owner` header. If the header is missing, the "" (empty string) is assigned as the item's owner during item creation. A mismatch will result in a "403 Forbidden" error. An authorized request may override this requirement, providing an administrative override. The method of authorization is not specified.

This endpoint allows for `GET`, and `DELETE` REST methods to interact with any
object that was created with the previous `PUT` request. An optional header
`X-Midt-Owner` can be sent with the request. All requests are validated by
comparing the secret stored with the requested record with the value sent in the
`X-Midt-Owner` header. If the header is missing, the "" (empty string) is
assigned as the item's owner during item creation. A mismatch will result in a
"403 Forbidden" error. An authorized request may override this requirement,
providing an administrative override. The method of authorization is not
specified.

An example response:
```json
Expand All @@ -130,15 +169,14 @@ An example response:
}
```


## Build

### Source

In order to build from the source, you need a working Go environment with
version 1.11 or greater. Find more information on the [Go website](https://golang.org/doc/install).

You can directly use `go get` to put the argus binary into your `GOPATH`:
You can directly use `go get` to put the Argus binary into your `GOPATH`:
```bash
go get github.com/xmidt-org/argus
```
Expand All @@ -156,12 +194,12 @@ make build
### Makefile

The Makefile has the following options you may find helpful:
* `make build`: builds the argus binary
* `make docker`: fetches all dependencies from source and builds an argus
docker image
* `make local-docker`: vendors dependencies and builds an argus docker image
* `make build`: builds the Argus binary
* `make docker`: fetches all dependencies from source and builds an Argus docker
image
* `make local-docker`: vendors dependencies and builds an Argus docker image
(recommended for local testing)
* `make test`: runs unit tests with coverage for argus
* `make test`: runs unit tests with coverage for Argus
* `make clean`: deletes previously-built binaries and object files

### RPM
Expand All @@ -180,31 +218,36 @@ command. Either option requires first getting the source code.

See [Makefile](#Makefile) on specifics of how to build the image that way.

If you'd like to build it without make, follow these instructions based on your use case:
If you'd like to build it without make, follow these instructions based on your
use case:

- Local testing
```bash
go mod vendor
docker build -t argus:local -f deploy/Dockerfile .
```
This allows you to test local changes to a dependency. For example, you can build
a argus image with the changes to an upcoming changes to [webpa-common](https://github.com/xmidt-org/webpa-common) by using the [replace](https://golang.org/ref/mod#go) directive in your go.mod file like so:
This allows you to test local changes to a dependency. For example, you can
build a Argus image with the changes to an upcoming changes to
[webpa-common](https://github.com/xmidt-org/webpa-common) by using the
[replace](https://golang.org/ref/mod#go) directive in your go.mod file like so:
```
replace github.com/xmidt-org/webpa-common v1.10.2-0.20200604164000-f07406b4eb63 => ../webpa-common
```
**Note:** if you omit `go mod vendor`, your build will fail as the path `../webpa-common` does not exist on the builder container.
**Note:** if you omit `go mod vendor`, your build will fail as the path
`../webpa-common` does not exist on the builder container.

- Building a specific version
```bash
git checkout v0.3.6
docker build -t argus:v0.3.6 -f deploy/Dockerfile .
```

**Additional Info:** If you'd like to stand up a XMiDT docker-compose cluster, read [this](https://github.com/xmidt-org/xmidt/blob/master/deploy/docker-compose/README.md).
**Additional Info:** If you'd like to stand up a XMiDT docker-compose cluster,
read [this](https://github.com/xmidt-org/xmidt/blob/master/deploy/docker-compose/README.md).

### Kubernetes

A helm chart can be used to deploy argus to kubernetes
A helm chart can be used to deploy Argus to kubernetes
```
helm install xmidt-argus deploy/helm/argus
```
Expand Down
9 changes: 2 additions & 7 deletions store/inputValidation.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ package store
import (
"encoding/json"
"regexp"
"strings"
"time"

"github.com/spf13/cast"
Expand Down Expand Up @@ -92,12 +91,8 @@ func (v *validItemUnmarshaler) UnmarshalJSON(data []byte) error {
return errDataFieldMissing
}

if !isIDValid(v.config.IDFormatRegex, v.item.ID) {
return errInvalidID
}

v.item.ID = strings.ToLower(v.item.ID)

// if we've gotten here, we've already validated the ID in the URL. The
// item ID just needs to match the ID from the URL.
if v.item.ID != v.id {
return errIDMismatch
}
Expand Down
5 changes: 2 additions & 3 deletions store/transport.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ import (
"net/http"
"regexp"
"sort"
"strings"
"time"

kithttp "github.com/go-kit/kit/transport/http"
Expand Down Expand Up @@ -105,7 +104,7 @@ func setItemRequestDecoder(config *transportConfig) kithttp.DecodeRequestFunc {
return func(ctx context.Context, r *http.Request) (interface{}, error) {
var (
URLVars = mux.Vars(r)
id = strings.ToLower(URLVars[idVarKey])
id = URLVars[idVarKey]
bucket = URLVars[bucketVarKey]
owner = r.Header.Get(ItemOwnerHeaderKey)
)
Expand Down Expand Up @@ -147,7 +146,7 @@ func getOrDeleteItemRequestDecoder(config *transportConfig) kithttp.DecodeReques
return func(ctx context.Context, r *http.Request) (interface{}, error) {
var (
URLVars = mux.Vars(r)
id = strings.ToLower(URLVars[idVarKey])
id = URLVars[idVarKey]
bucket = URLVars[bucketVarKey]
owner = r.Header.Get(ItemOwnerHeaderKey)
)
Expand Down
40 changes: 0 additions & 40 deletions store/transport_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ import (
"net/http"
"net/http/httptest"
"regexp"
"strings"
"testing"
"time"

Expand Down Expand Up @@ -112,19 +111,6 @@ func TestGetOrDeleteItemRequestDecoder(t *testing.T) {
},
},
},
{
Name: "Happy path. No owner. Normal mode. Uppercase ok",
URLVars: map[string]string{
"bucket": "california",
"id": strings.ToUpper(sfID),
},
ExpectedDecodedRequest: &getOrDeleteItemRequest{
key: model.Key{
Bucket: "california",
ID: sfID,
},
},
},
{
Name: "Happy path. Owner. Admin mode",
URLVars: map[string]string{
Expand Down Expand Up @@ -400,12 +386,6 @@ func TestSetItemRequestDecoder(t *testing.T) {
RequestBody: `{"id":"4b13653e5d6d611de5999ab0e7c0aa67e1d83d4cba8349a04da0a431fb27f74a", "data": {"x": 0, "y": 1, "z": 2}, "ttl": 3900}`,
ExpectedErr: errInvalidID,
},
{
Name: "Invalid Item ID",
URLVars: map[string]string{bucketVarKey: "variables", idVarKey: "4b13653e5d6d611de5999ab0e7c0aa67e1d83d4cba8349a04da0a431fb27f74b"},
RequestBody: `{"id":"notASha256HexDigest", "data": {"x": 0, "y": 1, "z": 2}, "ttl": 3900}`,
ExpectedErr: errInvalidID,
},
{
Name: "Invalid Bucket",
URLVars: map[string]string{bucketVarKey: "when-validation-gives-you-lemons!", idVarKey: "4b13653e5d6d611de5999ab0e7c0aa67e1d83d4cba8349a04da0a431fb27f74b"},
Expand Down Expand Up @@ -447,26 +427,6 @@ func TestSetItemRequestDecoder(t *testing.T) {
adminMode: true,
},
},
{
Name: "Alternative ID format",
URLVars: map[string]string{bucketVarKey: "variables", idVarKey: "4B13653E5D6D611DE5999AB0E7C0AA67E1D83D4CBA8349A04DA0A431FB27F74B"},
Owner: "mathematics",
RequestBody: `{"id":"4b13653e5d6d611de5999ab0e7c0aa67e1d83d4cba8349a04da0a431fb27f74b", "data": {"x": 0, "y": 1, "z": 2}, "ttl": 39}`,
ExpectedRequest: &setItemRequest{
item: OwnableItem{
Item: model.Item{
ID: "4b13653e5d6d611de5999ab0e7c0aa67e1d83d4cba8349a04da0a431fb27f74b",
Data: map[string]interface{}{"x": float64(0), "y": float64(1), "z": float64(2)},
TTL: aws.Int64(39),
},
Owner: "mathematics",
},
key: model.Key{
Bucket: "variables",
ID: "4b13653e5d6d611de5999ab0e7c0aa67e1d83d4cba8349a04da0a431fb27f74b",
},
},
},
}

decoder := setItemRequestDecoder(getTestTransportConfig())
Expand Down