Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: update @helia/ipns and dns config #18

Merged
merged 2 commits into from
Mar 14, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion packages/verified-fetch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ Note that you do not need to provide both a DNS-over-HTTPS and a DNS-over-JSON r

```typescript
import { createVerifiedFetch } from '@helia/verified-fetch'
import { dnsJsonOverHttps, dnsOverHttps } from '@helia/ipns/dns-resolvers'
import { dnsJsonOverHttps, dnsOverHttps } from '@multiformats/dns/resolvers'

const fetch = await createVerifiedFetch({
gateways: ['https://trustless-gateway.link'],
Expand All @@ -189,6 +189,29 @@ const fetch = await createVerifiedFetch({
})
```

## Example - Customizing DNS per-TLD resolvers

DNS resolvers can be configured to only service DNS queries for specific
TLDs:

```typescript
import { createVerifiedFetch } from '@helia/verified-fetch'
import { dnsJsonOverHttps, dnsOverHttps } from '@multiformats/dns/resolvers'

const fetch = await createVerifiedFetch({
gateways: ['https://trustless-gateway.link'],
routers: ['http://delegated-ipfs.dev'],
dnsResolvers: {
// this resolver will only be used for `.com` domains (note - this could
// also be an array of resolvers)
'com.': dnsJsonOverHttps('https://my-dns-resolver.example.com/dns-json'),
// this resolver will be used for everything else (note - this could
// also be an array of resolvers)
'.': dnsOverHttps('https://my-dns-resolver.example.com/dns-query')
}
})
```

### IPLD codec handling

IPFS supports several data formats (typically referred to as codecs) which are included in the CID. `@helia/verified-fetch` attempts to abstract away some of the details for easier consumption.
Expand Down
1 change: 1 addition & 0 deletions packages/verified-fetch/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@
"@libp2p/interface": "^1.1.4",
"@libp2p/kad-dht": "^12.0.8",
"@libp2p/peer-id": "^4.0.7",
"@multiformats/dns": "^1.0.2",
"cborg": "^4.0.9",
"hashlru": "^2.3.0",
"interface-blockstore": "^5.2.10",
Expand Down
57 changes: 49 additions & 8 deletions packages/verified-fetch/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@
*
* ```typescript
* import { createVerifiedFetch } from '@helia/verified-fetch'
* import { dnsJsonOverHttps, dnsOverHttps } from '@helia/ipns/dns-resolvers'
* import { dnsJsonOverHttps, dnsOverHttps } from '@multiformats/dns/resolvers'
*
* const fetch = await createVerifiedFetch({
* gateways: ['https://trustless-gateway.link'],
Expand All @@ -160,6 +160,29 @@
* })
* ```
*
* @example Customizing DNS per-TLD resolvers
*
* DNS resolvers can be configured to only service DNS queries for specific
* TLDs:
*
* ```typescript
* import { createVerifiedFetch } from '@helia/verified-fetch'
* import { dnsJsonOverHttps, dnsOverHttps } from '@multiformats/dns/resolvers'
*
* const fetch = await createVerifiedFetch({
* gateways: ['https://trustless-gateway.link'],
* routers: ['http://delegated-ipfs.dev'],
* dnsResolvers: {
* // this resolver will only be used for `.com` domains (note - this could
* // also be an array of resolvers)
* 'com.': dnsJsonOverHttps('https://my-dns-resolver.example.com/dns-json'),
* // this resolver will be used for everything else (note - this could
* // also be an array of resolvers)
* '.': dnsOverHttps('https://my-dns-resolver.example.com/dns-query')
* }
* })
* ```
*
* ### IPLD codec handling
*
* IPFS supports several data formats (typically referred to as codecs) which are included in the CID. `@helia/verified-fetch` attempts to abstract away some of the details for easier consumption.
Expand Down Expand Up @@ -569,10 +592,13 @@
import { trustlessGateway } from '@helia/block-brokers'
import { createHeliaHTTP } from '@helia/http'
import { delegatedHTTPRouting } from '@helia/routers'
import { dns } from '@multiformats/dns'
import { VerifiedFetch as VerifiedFetchClass } from './verified-fetch.js'
import type { Helia } from '@helia/interface'
import type { DNSResolver, IPNSRoutingEvents, ResolveDnsLinkProgressEvents, ResolveProgressEvents } from '@helia/ipns'
import type { ResolveDNSLinkProgressEvents } from '@helia/ipns'
import type { GetEvents } from '@helia/unixfs'
import type { DNSResolvers, DNS } from '@multiformats/dns'
import type { DNSResolver } from '@multiformats/dns/resolvers'
import type { CID } from 'multiformats/cid'
import type { ProgressEvent, ProgressOptions } from 'progress-events'

Expand Down Expand Up @@ -618,7 +644,7 @@ export interface CreateVerifiedFetchInit {
*
* @default [dnsJsonOverHttps('https://mozilla.cloudflare-dns.com/dns-query'),dnsJsonOverHttps('https://dns.google/resolve')]
*/
dnsResolvers?: DNSResolver[]
dnsResolvers?: DNSResolver[] | DNSResolvers
}

export interface CreateVerifiedFetchOptions {
Expand Down Expand Up @@ -651,7 +677,7 @@ export type BubbledProgressEvents =
// unixfs
GetEvents |
// ipns
ResolveProgressEvents | ResolveDnsLinkProgressEvents | IPNSRoutingEvents
ResolveDNSLinkProgressEvents

export type VerifiedFetchProgressEvents =
ProgressEvent<'verified-fetch:request:start', CIDDetail> |
Expand All @@ -674,20 +700,19 @@ export interface VerifiedFetchInit extends RequestInit, ProgressOptions<BubbledP
* Create and return a Helia node
*/
export async function createVerifiedFetch (init?: Helia | CreateVerifiedFetchInit, options?: CreateVerifiedFetchOptions): Promise<VerifiedFetch> {
let dnsResolvers: DNSResolver[] | undefined
if (!isHelia(init)) {
dnsResolvers = init?.dnsResolvers
init = await createHeliaHTTP({
blockBrokers: [
trustlessGateway({
gateways: init?.gateways
})
],
routers: (init?.routers ?? ['https://delegated-ipfs.dev']).map((routerUrl) => delegatedHTTPRouting(routerUrl))
routers: (init?.routers ?? ['https://delegated-ipfs.dev']).map((routerUrl) => delegatedHTTPRouting(routerUrl)),
dns: createDns(init?.dnsResolvers)
})
}

const verifiedFetchInstance = new VerifiedFetchClass({ helia: init }, { dnsResolvers, ...options })
const verifiedFetchInstance = new VerifiedFetchClass({ helia: init }, options)
async function verifiedFetch (resource: Resource, options?: VerifiedFetchInit): Promise<Response> {
return verifiedFetchInstance.fetch(resource, options)
}
Expand All @@ -707,3 +732,19 @@ function isHelia (obj: any): obj is Helia {
obj?.stop != null &&
obj?.start != null
}

function createDns (resolvers?: DNSResolver[] | DNSResolvers): DNS | undefined {
if (resolvers == null) {
return
}

if (Array.isArray(resolvers)) {
return dns({
resolvers: {
'.': resolvers
}
})
}

return dns({ resolvers })
}
4 changes: 2 additions & 2 deletions packages/verified-fetch/src/utils/parse-resource.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { CID } from 'multiformats/cid'
import { parseUrlString } from './parse-url-string.js'
import type { ParsedUrlStringResults } from './parse-url-string.js'
import type { Resource } from '../index.js'
import type { IPNS, IPNSRoutingEvents, ResolveDnsLinkProgressEvents, ResolveProgressEvents } from '@helia/ipns'
import type { IPNS, IPNSRoutingEvents, ResolveDNSLinkProgressEvents, ResolveProgressEvents } from '@helia/ipns'
import type { ComponentLogger } from '@libp2p/interface'
import type { ProgressOptions } from 'progress-events'

Expand All @@ -11,7 +11,7 @@ export interface ParseResourceComponents {
logger: ComponentLogger
}

export interface ParseResourceOptions extends ProgressOptions<ResolveProgressEvents | IPNSRoutingEvents | ResolveDnsLinkProgressEvents> {
export interface ParseResourceOptions extends ProgressOptions<ResolveProgressEvents | IPNSRoutingEvents | ResolveDNSLinkProgressEvents> {

}
/**
Expand Down
6 changes: 3 additions & 3 deletions packages/verified-fetch/src/utils/parse-url-string.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { peerIdFromString } from '@libp2p/peer-id'
import { CID } from 'multiformats/cid'
import { TLRU } from './tlru.js'
import type { RequestFormatShorthand } from '../types.js'
import type { IPNS, IPNSRoutingEvents, ResolveDnsLinkProgressEvents, ResolveProgressEvents, ResolveResult } from '@helia/ipns'
import type { IPNS, ResolveDNSLinkProgressEvents, ResolveResult } from '@helia/ipns'
import type { ComponentLogger } from '@libp2p/interface'
import type { ProgressOptions } from 'progress-events'

Expand All @@ -13,7 +13,7 @@ export interface ParseUrlStringInput {
ipns: IPNS
logger: ComponentLogger
}
export interface ParseUrlStringOptions extends ProgressOptions<ResolveProgressEvents | IPNSRoutingEvents | ResolveDnsLinkProgressEvents> {
export interface ParseUrlStringOptions extends ProgressOptions<ResolveDNSLinkProgressEvents> {

}

Expand Down Expand Up @@ -134,7 +134,7 @@ export async function parseUrlString ({ urlString, ipns, logger }: ParseUrlStrin
log.trace('Attempting to resolve DNSLink for %s', decodedDnsLinkLabel)

try {
resolveResult = await ipns.resolveDns(decodedDnsLinkLabel, { onProgress: options?.onProgress })
resolveResult = await ipns.resolveDNSLink(decodedDnsLinkLabel, { onProgress: options?.onProgress })
cid = resolveResult?.cid
resolvedPath = resolveResult?.path
log.trace('resolved %s to %c', decodedDnsLinkLabel, cid)
Expand Down
11 changes: 3 additions & 8 deletions packages/verified-fetch/src/verified-fetch.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import { car } from '@helia/car'
import { ipns as heliaIpns, type DNSResolver, type IPNS } from '@helia/ipns'
import { dnsJsonOverHttps } from '@helia/ipns/dns-resolvers'
import { ipns as heliaIpns, type IPNS } from '@helia/ipns'
import { unixfs as heliaUnixFs, type UnixFS as HeliaUnixFs, type UnixFSStats } from '@helia/unixfs'
import * as ipldDagCbor from '@ipld/dag-cbor'
import * as ipldDagJson from '@ipld/dag-json'
Expand Down Expand Up @@ -29,6 +28,7 @@ import type { CIDDetail, ContentTypeParser, Resource, VerifiedFetchInit as Verif
import type { RequestFormatShorthand } from './types.js'
import type { Helia } from '@helia/interface'
import type { AbortOptions, Logger, PeerId } from '@libp2p/interface'
import type { DNSResolver } from '@multiformats/dns/resolvers'
import type { UnixFSEntry } from 'ipfs-unixfs-exporter'
import type { CID } from 'multiformats/cid'

Expand Down Expand Up @@ -126,12 +126,7 @@ export class VerifiedFetch {
constructor ({ helia, ipns, unixfs }: VerifiedFetchComponents, init?: VerifiedFetchInit) {
this.helia = helia
this.log = helia.logger.forComponent('helia:verified-fetch')
this.ipns = ipns ?? heliaIpns(helia, {
resolvers: init?.dnsResolvers ?? [
dnsJsonOverHttps('https://mozilla.cloudflare-dns.com/dns-query'),
dnsJsonOverHttps('https://dns.google/resolve')
]
})
this.ipns = ipns ?? heliaIpns(helia)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

this.unixfs = unixfs ?? heliaUnixFs(helia)
this.contentTypeParser = init?.contentTypeParser
this.log.trace('created VerifiedFetch instance')
Expand Down
32 changes: 26 additions & 6 deletions packages/verified-fetch/test/custom-dns-resolvers.spec.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { stop } from '@libp2p/interface'
import { dns, RecordType } from '@multiformats/dns'
import { expect } from 'aegir/chai'
import Sinon from 'sinon'
import { createVerifiedFetch } from '../src/index.js'
Expand Down Expand Up @@ -30,23 +31,42 @@ describe('custom dns-resolvers', () => {
await expect(fetch('ipns://some-non-cached-domain.com')).to.eventually.be.rejected.with.property('errors')

expect(customDnsResolver.callCount).to.equal(1)
expect(customDnsResolver.getCall(0).args).to.deep.equal(['some-non-cached-domain.com', { onProgress: undefined }])
expect(customDnsResolver.getCall(0).args).to.deep.equal(['_dnslink.some-non-cached-domain.com', {
onProgress: undefined,
types: [
RecordType.TXT
]
}])
})

it('is used when passed to VerifiedFetch', async () => {
const customDnsResolver = Sinon.stub()
const customDnsResolver = Sinon.stub().withArgs('_dnslink.some-non-cached-domain2.com').resolves({
Answer: [{
data: 'dnslink=/ipfs/QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg'
}]
})

customDnsResolver.returns(Promise.resolve('/ipfs/QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg'))
await stop(helia)
helia = await createHelia({
dns: dns({
resolvers: {
'.': customDnsResolver
}
})
})

const verifiedFetch = new VerifiedFetch({
helia
}, {
dnsResolvers: [customDnsResolver]
})
// error of walking the CID/dag because we haven't actually added the block to the blockstore
await expect(verifiedFetch.fetch('ipns://some-non-cached-domain2.com')).to.eventually.be.rejected.with.property('errors').that.has.lengthOf(0)

expect(customDnsResolver.callCount).to.equal(1)
expect(customDnsResolver.getCall(0).args).to.deep.equal(['some-non-cached-domain2.com', { onProgress: undefined }])
expect(customDnsResolver.getCall(0).args).to.deep.equal(['_dnslink.some-non-cached-domain2.com', {
onProgress: undefined,
types: [
RecordType.TXT
]
}])
})
})
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
import { Helia as HeliaClass } from '@helia/utils'
import { MemoryBlockstore } from 'blockstore-core'
import { MemoryDatastore } from 'datastore-core'
import type { HeliaHTTPInit } from '@helia/http'
import type { Helia } from '@helia/interface'

export async function createHelia (): Promise<Helia> {
export async function createHelia (init: Partial<HeliaHTTPInit> = {}): Promise<Helia> {
const datastore = new MemoryDatastore()
const blockstore = new MemoryBlockstore()

const helia = new HeliaClass({
datastore,
blockstore,
blockBrokers: [],
routers: []
routers: [],
...init
})

await helia.start()
Expand Down
2 changes: 1 addition & 1 deletion packages/verified-fetch/test/parse-resource.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ describe('parseResource', () => {
const shouldNotBeCalled2 = sinon.stub().throws(new Error('should not be called'))
const { cid, path, query } = await parseResource(testCID, {
ipns: stubInterface<IPNS>({
resolveDns: shouldNotBeCalled1,
resolveDNSLink: shouldNotBeCalled1,
resolve: shouldNotBeCalled2
}),
logger: defaultLogger()
Expand Down
Loading
Loading