Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat!: use Helia's blockBroker interface #406

Merged
merged 10 commits into from
Nov 29, 2023
Merged

feat!: use Helia's blockBroker interface #406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
6f29b4d
deps: update helia
SgtPooki Nov 29, 2023
73cfbe6
feat: use new Helia blockBroker interface
SgtPooki Nov 29, 2023
c2fa4ad
deps: update 'ipld|multi' deps
SgtPooki Nov 29, 2023
dd05dcd
deps: update 'ipfs' deps
SgtPooki Nov 29, 2023
57fe725
chore: fix lint after eslint-config-ipfs update
SgtPooki Nov 29, 2023
e18328b
feat!: remove kubo-rpc-client entirely
SgtPooki Nov 29, 2023
3b5f578
chore: apiOpts -> kuboGatewayOptions
SgtPooki Nov 29, 2023
5a035a1
fix: types and errors
SgtPooki Nov 29, 2023
d05498b
fix: bigint use in ObjectInfo.jsx
SgtPooki Nov 29, 2023
252c6fb
fix: tests and vitest config
SgtPooki Nov 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions dev/devPage.jsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,9 +91,10 @@ const PageRenderer = connect(
(props) => {
const Page = props?.route
const { embed } = props.queryObject
const { doInitHelia } = props
useEffect(() => {
props.doInitHelia()
}, [props])
doInitHelia()
}, [doInitHelia])
Comment on lines +94 to +97
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this fixes an issue where doInitHelia was being called on every page load.


return (
<>
Expand Down
17,617 changes: 10,151 additions & 7,466 deletions package-lock.json
View file
Open in desktop

Large diffs are not rendered by default.

37 changes: 18 additions & 19 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,29 +118,28 @@
"storybook:build": "storybook build"
},
"dependencies": {
"@chainsafe/libp2p-gossipsub": "^8.0.0",
"@chainsafe/libp2p-noise": "^12.0.0",
"@chainsafe/libp2p-yamux": "^4.0.2",
"@ipld/car": "^5.1.1",
"@ipld/dag-cbor": "^9.0.1",
"@ipld/dag-json": "^10.0.1",
"@ipld/dag-pb": "^4.0.3",
"@libp2p/delegated-content-routing": "^4.0.3",
"@libp2p/delegated-peer-routing": "^4.0.6",
"@chainsafe/libp2p-gossipsub": "^10.1.1",
"@chainsafe/libp2p-noise": "^13.0.5",
"@chainsafe/libp2p-yamux": "^5.0.3",
"@ipld/car": "^5.2.4",
"@ipld/dag-cbor": "^9.0.6",
"@ipld/dag-json": "^10.1.5",
"@ipld/dag-pb": "^4.0.6",
"@libp2p/delegated-content-routing": "^4.0.11",
"@libp2p/delegated-peer-routing": "^4.0.14",
"@libp2p/ipni-content-routing": "^1.0.1",
"@loadable/component": "^5.14.1",
"@tableflip/react-inspector": "^2.3.0",
"cytoscape": "^3.18.1",
"cytoscape-dagre": "^2.3.2",
"filesize": "^6.1.0",
"helia": "^1.1.2",
"ipfs-unixfs": "^4.0.1",
"helia": "^2.1.0",
"ipfs-unixfs": "^11.1.0",
"ipld-git": "^0.6.6",
"ipld-raw": "7.0.0",
"kubo-rpc-client": "^3.0.1",
"libp2p": "^0.45.1",
"libp2p": "^0.46.21",
"milliseconds": "^1.0.3",
"multiformats": "^11.0.2",
"multiformats": "^12.1.3",
"react-joyride": "^2.3.0",
"stream-to-it": "^0.2.4",
"web-encoding": "^1.1.5"
Expand All @@ -157,9 +156,9 @@
"@esbuild-plugins/esm-externals": "^0.1.2",
"@esbuild-plugins/node-globals-polyfill": "^0.2.3",
"@esbuild-plugins/node-modules-polyfill": "^0.2.2",
"@helia/interface": "^1.0.0",
"@libp2p/bootstrap": "^8.0.0",
"@libp2p/websockets": "^6.0.1",
"@helia/interface": "^2.1.0",
"@libp2p/bootstrap": "^9.0.12",
"@libp2p/websockets": "^7.0.13",
"@rollup/plugin-commonjs": "^25.0.0",
"@rollup/plugin-json": "^6.0.0",
"@rollup/plugin-node-resolve": "^15.0.2",
Expand Down Expand Up @@ -197,7 +196,7 @@
"dag-jose": "^4.0.0",
"datastore-core": "^9.1.1",
"eslint": "^8.41.0",
"eslint-config-ipfs": "^4.0.3",
"eslint-config-ipfs": "^6.0.0",
"eslint-config-react-app": "^7.0.1",
"eslint-config-standard": "^17.0.0",
"eslint-config-standard-react": "^13.0.0",
Expand All @@ -217,7 +216,7 @@
"interface-datastore": "^8.2.0",
"intl-messageformat": "^10.3.5",
"ipfs-core": "^0.18.0",
"ipfs-css": "^1.3.0",
"ipfs-css": "^1.4.0",
"jsdom": "^22.1.0",
"npm-run-all": "^4.1.5",
"prop-types": "^15.8.1",
Expand Down
2 changes: 1 addition & 1 deletion src/bundles/explore.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ const makeBundle = () => {
localPath,
nodes,
pathBoundaries
} = await resolveIpldPath(store.selectHelia(), store.selectKuboClient(), cid, rest)
} = await resolveIpldPath(store.selectHelia(), cid, rest)

return {
path,
Expand Down
34 changes: 14 additions & 20 deletions src/bundles/helia.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,23 @@
/* globals globalThis */
import { type Helia } from '@helia/interface'
import { create as createKuboClient, type IPFSHTTPClient } from 'kubo-rpc-client'

import initHelia from '../lib/init-helia'
import initHelia from '../lib/init-helia.js'
import type { KuboGatewayOptions } from '../types.d.js'

interface HeliaBundleState {
apiOpts: Record<string, string>
kuboGatewayOptions: KuboGatewayOptions
instance: Helia | null
error: Error | null
kuboClient: IPFSHTTPClient | null
}

const defaultState: HeliaBundleState = {
apiOpts: {
kuboGatewayOptions: {
host: '127.0.0.1',
port: '5001',
port: '8080',
SgtPooki marked this conversation as resolved.
Show resolved Hide resolved
protocol: 'http'
},
instance: null,
error: null,
kuboClient: null
error: null
}

function getUserOpts (key: string): Record<string, unknown> {
Expand All @@ -42,8 +40,7 @@
if (type === 'HELIA_INIT_FINISHED') {
return Object.assign({}, state, {
instance: payload.instance ?? state.instance,
kuboClient: payload.kuboClient ?? state.kuboClient,
apiOpts: payload.apiOpts ?? state.apiOpts,
kuboGatewayOptions: payload.kuboGatewayOptions ?? state.kuboGatewayOptions,
error: null
})
}
Expand All @@ -55,7 +52,6 @@
return state
},

selectKuboClient: ({ helia }: { helia: HeliaBundleState }): IPFSHTTPClient | null => helia.kuboClient,
selectHelia: ({ helia }: { helia: HeliaBundleState }) => helia.instance,

selectHeliaReady: ({ helia }: { helia: HeliaBundleState }) => helia.instance !== null,
Expand All @@ -66,28 +62,26 @@
return identifyService?.host?.agentVersion.split(' ')[0] ?? 'null'
},

doInitHelia: () => async ({ dispatch, getState }: any) => {

Check warning on line 65 in src/bundles/helia.ts

View workflow job for this annotation

GitHub Actions / js-test-and-release / check 

Unexpected any. Specify a different type

Check warning on line 65 in src/bundles/helia.ts

View workflow job for this annotation

GitHub Actions / js-test-and-release / check 

Unexpected any. Specify a different type

Check warning on line 65 in src/bundles/helia.ts

View workflow job for this annotation

GitHub Actions / js-test-and-release / test-webkit (macos-latest, lts/*) 

Unexpected any. Specify a different type

Check warning on line 65 in src/bundles/helia.ts

View workflow job for this annotation

GitHub Actions / js-test-and-release / test-webkit-webworker (macos-latest, lts/*) 

Unexpected any. Specify a different type
dispatch({ type: 'HELIA_INIT_STARTED' })

const apiOpts = Object.assign(
const kuboGatewayOptions = Object.assign(
{},
getState().helia.apiOpts,
getUserOpts('ipfsApi')
getState().helia.kuboGatewayOptions,
getUserOpts('kuboGateway')
)
// TRY helia!

try {
console.info(
"🎛️ Customise your kubo-rpc-client opts by setting an `ipfsApi` value in localStorage. e.g. localStorage.setItem('ipfsApi', JSON.stringify({port: '1337'}))"
"🎛️ Customise your Kubo gateway opts by setting an `kuboGateway` value in localStorage. e.g. localStorage.setItem('kuboGateway', JSON.stringify({port: '1337'}))"
)
const kuboClient = createKuboClient(apiOpts)
console.time('HELIA_INIT')
const helia = await initHelia(kuboClient)
const helia = await initHelia(kuboGatewayOptions)
console.timeEnd('HELIA_INIT')
return dispatch({
type: 'HELIA_INIT_FINISHED',
payload: {
apiOpts,
kuboClient,
kuboGatewayOptions,
instance: helia,
provider: 'helia'
}
Expand Down
36 changes: 34 additions & 2 deletions src/components/object-info/ObjectInfo.jsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,39 @@ const DagNodeIcon = ({ type, ...props }) => (
</svg>
)

/**
* replace bigint or other non-JSON-serializable values with appropriate values for the react-inspector
* Note that data for some blocks (e.g. bafyreicnokmhmrnlp2wjhyk2haep4tqxiptwfrp2rrs7rzq7uk766chqvq) currently do not
* look like NormalizedDagNode['data']
*
* @param {import('../../types').NormalizedDagNode['data']} data
*/
const getObjectInspectorData = (data) => {
if (data == null) return data
if (data.blockSizes != null) {
data.blockSizes = data.blockSizes.map(Number)
}
return data
}
Comment on lines +82 to +95
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needed since the unixfs update uses bigint now.


/**
* @param {object} props
* @param {import('react-i18next').TFunction} props.t
* @param {boolean} props.tReady
* @param {string} props.className
* @param {string} props.type
* @param {string} props.cid
* @param {string} props.localPath
* @param {bigint} props.size
* @param {import('../../types').NormalizedDagNode['data']} props.data
* @param {object[]} props.links
* @param {string} props.format
* @param {Function} props.onLinkClick
* @param {string} props.gatewayUrl
* @param {string} props.publicGatewayUrl
*/
const ObjectInfo = ({ t, tReady, className, type, cid, localPath, size, data, links, format, onLinkClick, gatewayUrl, publicGatewayUrl, ...props }) => {
Comment on lines +97 to 113
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some types

if (!tReady) return null
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is always true in testing, but may not always be in prod.

const isUnixFs = format === 'unixfs' && data.type && ['directory', 'file'].some(x => x === data.type)
let nodeStyleType = type

Expand Down Expand Up @@ -130,7 +162,7 @@ const ObjectInfo = ({ t, tReady, className, type, cid, localPath, size, data, li
<div className='dt dt--fixed pt2'>
{/* eslint-disable-next-line jsx-a11y/label-has-associated-control */}
<label className='dtc silver tracked ttu f7' style={{ width: 48 }}>Size</label>
<div className='dtc truncate charcoal monospace'>{humansize(size)}</div>
<div className='dtc truncate charcoal monospace'>{humansize(Number(size))}</div>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bigint conversion

</div>
)}
<div className='dt dt--fixed pt2'>
Expand All @@ -151,7 +183,7 @@ const ObjectInfo = ({ t, tReady, className, type, cid, localPath, size, data, li
? null
: (
<div className='pa3 mt2 bg-white f5 nl3 nr3 mh0-l'>
<ObjectInspector showMaxKeys={100} data={data} theme={objectInspectorTheme} expandPaths={toExpandPathsNotation(localPath)} />
<ObjectInspector showMaxKeys={100} data={getObjectInspectorData(data)} theme={objectInspectorTheme} expandPaths={toExpandPathsNotation(localPath)} />
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bigint conversion

</div>
)}
</div>
Expand Down
10 changes: 5 additions & 5 deletions src/global.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,16 @@
*/
declare module 'stream-to-it' {
interface toIterable {
source: <T>(stream: ReadableStream<T>) => AsyncIterable<T>
source<T>(stream: ReadableStream<T>): AsyncIterable<T>
}
const toIterable: toIterable
export default toIterable
}

interface OldIpldFormat {
util: {
serialize: (obj: unknown) => Promise<Uint8Array>
deserialize: (bytes: Uint8Array) => Promise<unknown>
serialize(obj: unknown): Promise<Uint8Array>
deserialize(bytes: Uint8Array): Promise<unknown>

codec: number
defaultHashAlg: number
Expand All @@ -21,8 +21,8 @@ interface OldIpldFormat {
}
codec: number
resolver: {
resolve: (bytes: Uint8Array, path: string) => Promise<unknown>
tree: (bytes: Uint8Array) => Promise<unknown>
resolve(bytes: Uint8Array, path: string): Promise<unknown>
tree(bytes: Uint8Array): Promise<unknown>
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eslint-config-ipfs auto lint updates

defaultHashAlg: number
}
Expand Down
2 changes: 0 additions & 2 deletions src/lib/codec-importer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,12 @@ export default async function codecImporter<T extends CodecDataTypes = CodecData
case 'dag-cbor':
return import('@ipld/dag-cbor')
case 'dag-pb':
// @ts-expect-error - return types need normalizing
return import('@ipld/dag-pb')
case 'git-raw':
return {
decode: (await import('ipld-git')).default.util.deserialize
}
case 'raw':
// @ts-expect-error - return types need normalizing
Comment on lines -16 to -23
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yay, better types.

return import('multiformats/codecs/raw')
case 'json':
return import('multiformats/codecs/json')
Expand Down
4 changes: 2 additions & 2 deletions src/lib/get-codec-for-cid.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ import { ensureLeadingSlash } from './helpers'
import type { ResolveType } from '../types'

interface CodecWrapper<DecodedType = any> {
decode: (bytes: Uint8Array) => DecodedType
resolve: (path: string, bytes: Uint8Array) => Promise<ResolveType<DecodedType>>
decode(bytes: Uint8Array): DecodedType
resolve(path: string, bytes: Uint8Array): Promise<ResolveType<DecodedType>>
}
Comment on lines 10 to 13
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eslint-config-ipfs auto lint updates


interface DecodeFn<T = any> {
Expand Down
Loading
Loading