-
Notifications
You must be signed in to change notification settings - Fork 101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature: Sync with meilisearch in bun runtime #223
Merged
loks0n
merged 8 commits into
appwrite:main
from
FreSauce:feat-implement-sync-with-meilisearch-bun
Nov 16, 2023
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
f3fd446
sync with meilisearch in bun runtime
FreSauce 6d6d0d2
Merge branch 'appwrite:main' into feat-implement-sync-with-meilisearc…
FreSauce 4269693
removed comments and changed env
FreSauce 06ad690
Merge branch 'appwrite:main' into feat-implement-sync-with-meilisearc…
FreSauce 8714adc
fixed process.env to Bun.env and changed comment style
FreSauce aa41fa7
Merge branch 'appwrite:main' into feat-implement-sync-with-meilisearc…
FreSauce daac51f
fixed readme and package.json
FreSauce 3639d16
Merge branch 'appwrite:main' into feat-implement-sync-with-meilisearc…
FreSauce File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,145 @@ | ||
### Node ### | ||
# Logs | ||
logs | ||
*.log | ||
npm-debug.log* | ||
yarn-debug.log* | ||
yarn-error.log* | ||
lerna-debug.log* | ||
.pnpm-debug.log* | ||
|
||
# Diagnostic reports (https://nodejs.org/api/report.html) | ||
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json | ||
|
||
# Runtime data | ||
pids | ||
*.pid | ||
*.seed | ||
*.pid.lock | ||
|
||
# Directory for instrumented libs generated by jscoverage/JSCover | ||
lib-cov | ||
|
||
# Coverage directory used by tools like istanbul | ||
coverage | ||
*.lcov | ||
|
||
# nyc test coverage | ||
.nyc_output | ||
|
||
# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files) | ||
.grunt | ||
|
||
# Bower dependency directory (https://bower.io/) | ||
bower_components | ||
|
||
# node-waf configuration | ||
.lock-wscript | ||
|
||
# Compiled binary addons (https://nodejs.org/api/addons.html) | ||
build/Release | ||
|
||
# Dependency directories | ||
node_modules/ | ||
jspm_packages/ | ||
|
||
# Snowpack dependency directory (https://snowpack.dev/) | ||
web_modules/ | ||
|
||
# TypeScript cache | ||
*.tsbuildinfo | ||
|
||
# Optional npm cache directory | ||
.npm | ||
|
||
# Optional eslint cache | ||
.eslintcache | ||
|
||
# Optional stylelint cache | ||
.stylelintcache | ||
|
||
# Microbundle cache | ||
.rpt2_cache/ | ||
.rts2_cache_cjs/ | ||
.rts2_cache_es/ | ||
.rts2_cache_umd/ | ||
|
||
# Optional REPL history | ||
.node_repl_history | ||
|
||
# Output of 'npm pack' | ||
*.tgz | ||
|
||
# Yarn Integrity file | ||
.yarn-integrity | ||
|
||
# dotenv environment variable files | ||
.env | ||
.env.development.local | ||
.env.test.local | ||
.env.production.local | ||
.env.local | ||
|
||
# parcel-bundler cache (https://parceljs.org/) | ||
.cache | ||
.parcel-cache | ||
|
||
# Next.js build output | ||
.next | ||
out | ||
|
||
# Nuxt.js build / generate output | ||
.nuxt | ||
dist | ||
|
||
# Gatsby files | ||
.cache/ | ||
# Comment in the public line in if your project uses Gatsby and not Next.js | ||
# https://nextjs.org/blog/next-9-1#public-directory-support | ||
# public | ||
|
||
# vuepress build output | ||
.vuepress/dist | ||
|
||
# vuepress v2.x temp and cache directory | ||
.temp | ||
|
||
# Docusaurus cache and generated files | ||
.docusaurus | ||
|
||
# Serverless directories | ||
.serverless/ | ||
|
||
# FuseBox cache | ||
.fusebox/ | ||
|
||
# DynamoDB Local files | ||
.dynamodb/ | ||
|
||
# TernJS port file | ||
.tern-port | ||
|
||
# Stores VSCode versions used for testing VSCode extensions | ||
.vscode-test | ||
|
||
# yarn v2 | ||
.yarn/cache | ||
.yarn/unplugged | ||
.yarn/build-state.yml | ||
.yarn/install-state.gz | ||
.pnp.* | ||
|
||
### Node Patch ### | ||
# Serverless Webpack directories | ||
.webpack/ | ||
|
||
# Optional stylelint cache | ||
|
||
# SvelteKit build / generate output | ||
.svelte-kit | ||
|
||
# End of https://www.toptal.com/developers/gitignore/api/node | ||
|
||
# OS | ||
## Mac | ||
.DS_Store |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,106 @@ | ||
# ⚡ Bun Sync with Meilisearch Function | ||
|
||
Syncs documents in an Appwrite database collection to a Meilisearch index. | ||
|
||
## 🧰 Usage | ||
|
||
### GET / | ||
|
||
Returns HTML page where search can be performed to test the indexing. | ||
|
||
### POST / | ||
|
||
Triggers indexing of the Appwrite database collection to Meilisearch. | ||
|
||
**Response** | ||
|
||
Sample `204` Response: No content. | ||
|
||
## ⚙️ Configuration | ||
|
||
| Setting | Value | | ||
| ----------------- | ------------- | | ||
| Runtime | Bun (1.0) | | ||
| Entrypoint | `src/main.ts` | | ||
| Build Commands | `bun install` | | ||
| Permissions | `any` | | ||
| Timeout (Seconds) | 15 | | ||
|
||
## 🔒 Environment Variables | ||
|
||
### APPWRITE_API_KEY | ||
|
||
API Key to talk to Appwrite backend APIs. | ||
|
||
| Question | Answer | | ||
| ------------- | -------------------------------------------------------------------------------------------------- | | ||
| Required | Yes | | ||
| Sample Value | `d1efb...aec35` | | ||
| Documentation | [Appwrite: Getting Started for Server](https://appwrite.io/docs/getting-started-for-server#apiKey) | | ||
|
||
### APPWRITE_DATABASE_ID | ||
|
||
The ID of the Appwrite database that contains the collection to sync. | ||
|
||
| Question | Answer | | ||
| ------------- | --------------------------------------------------------- | | ||
| Required | Yes | | ||
| Sample Value | `612a3...5b6c9` | | ||
| Documentation | [Appwrite: Databases](https://appwrite.io/docs/databases) | | ||
|
||
### APPWRITE_COLLECTION_ID | ||
|
||
The ID of the collection in the Appwrite database to sync. | ||
|
||
| Question | Answer | | ||
| ------------- | ------------------------------------------------------------- | | ||
| Required | Yes | | ||
| Sample Value | `7c3e8...2a9f1` | | ||
| Documentation | [Appwrite: Collections](https://appwrite.io/docs/databases#collection) | | ||
|
||
### APPWRITE_ENDPOINT | ||
|
||
The URL endpoint of the Appwrite server. If not provided, it defaults to the Appwrite Cloud server: `https://cloud.appwrite.io/v1`. | ||
|
||
| Question | Answer | | ||
| ------------ | ------------------------------ | | ||
| Required | No | | ||
| Sample Value | `https://cloud.appwrite.io/v1` | | ||
|
||
### MEILISEARCH_ENDPOINT | ||
|
||
The host URL of the Meilisearch server. | ||
|
||
| Question | Answer | | ||
| ------------ | ----------------------- | | ||
| Required | Yes | | ||
| Sample Value | `http://127.0.0.1:7700` | | ||
|
||
### MEILISEARCH_ADMIN_API_KEY | ||
|
||
The admin API key for Meilisearch. | ||
|
||
| Question | Answer | | ||
| ------------- | ------------------------------------------------------------------------ | | ||
| Required | Yes | | ||
| Sample Value | `masterKey1234` | | ||
| Documentation | [Meilisearch: API Keys](https://docs.meilisearch.com/reference/api/keys) | | ||
|
||
### MEILISEARCH_INDEX_NAME | ||
|
||
Name of the Meilisearch index to which the documents will be synchronized. | ||
|
||
| Question | Answer | | ||
| ------------ | ---------- | | ||
| Required | Yes | | ||
| Sample Value | `my_index` | | ||
|
||
### MEILISEARCH_SEARCH_API_KEY | ||
|
||
API Key for Meilisearch search operations. | ||
|
||
| Question | Answer | | ||
| ------------- | ------------------------------------------------------------------------ | | ||
| Required | Yes | | ||
| Sample Value | `searchKey1234` | | ||
| Documentation | [Meilisearch: API Keys](https://docs.meilisearch.com/reference/api/keys) | |
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
declare module "bun" { | ||
interface Env { | ||
APPWRITE_ENDPOINT?: string; | ||
APPWRITE_API_KEY: string; | ||
APPWRITE_FUNCTION_PROJECT_ID: string; | ||
APPWRITE_DATABASE_ID: string; | ||
APPWRITE_COLLECTION_ID: string; | ||
MEILISEARCH_ENDPOINT: string; | ||
MEILISEARCH_INDEX_NAME: string; | ||
MEILISEARCH_ADMIN_API_KEY: string; | ||
MEILISEARCH_SEARCH_API_KEY: string; | ||
} | ||
} | ||
|
||
export { }; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
{ | ||
"name": "sync-with-meilisearch", | ||
"version": "1.0.0", | ||
"description": "", | ||
"main": "src/main.ts", | ||
"type": "module", | ||
"scripts": { | ||
"format": "prettier --write ." | ||
}, | ||
"dependencies": { | ||
"bun-types": "^1.0.7", | ||
"meilisearch": "^0.35.0", | ||
"node-appwrite": "^11.0.0", | ||
"path": "^0.12.7", | ||
"url": "^0.11.3" | ||
}, | ||
"devDependencies": { | ||
"prettier": "^3.0.0" | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,72 @@ | ||
import { Client, Databases, Query } from 'node-appwrite'; | ||
import { getStaticFile, interpolate, throwIfMissing } from './utils'; | ||
import { MeiliSearch } from 'meilisearch'; | ||
|
||
export default async ({ req, res, log }) => { | ||
throwIfMissing(Bun.env, [ | ||
'APPWRITE_API_KEY', | ||
'APPWRITE_DATABASE_ID', | ||
'APPWRITE_COLLECTION_ID', | ||
'MEILISEARCH_ENDPOINT', | ||
'MEILISEARCH_INDEX_NAME', | ||
'MEILISEARCH_ADMIN_API_KEY', | ||
'MEILISEARCH_SEARCH_API_KEY', | ||
]); | ||
|
||
if (req.method === 'GET') { | ||
const html = interpolate(await getStaticFile('index.html'), { | ||
MEILISEARCH_ENDPOINT: Bun.env.MEILISEARCH_ENDPOINT, | ||
MEILISEARCH_INDEX_NAME: Bun.env.MEILISEARCH_INDEX_NAME, | ||
MEILISEARCH_SEARCH_API_KEY: Bun.env.MEILISEARCH_SEARCH_API_KEY, | ||
}); | ||
|
||
return res.send(html, 200, { 'Content-Type': 'text/html; charset=utf-8' }); | ||
} | ||
|
||
const client = new Client() | ||
.setEndpoint( | ||
Bun.env.APPWRITE_ENDPOINT ?? 'https://cloud.appwrite.io/v1' | ||
) | ||
.setProject(Bun.env.APPWRITE_FUNCTION_PROJECT_ID) | ||
.setKey(Bun.env.APPWRITE_API_KEY); | ||
|
||
const databases = new Databases(client); | ||
|
||
const meilisearch = new MeiliSearch({ | ||
host: Bun.env.MEILISEARCH_ENDPOINT, | ||
apiKey: Bun.env.MEILISEARCH_ADMIN_API_KEY, | ||
}); | ||
|
||
const index = meilisearch.index(Bun.env.MEILISEARCH_INDEX_NAME); | ||
|
||
let cursor = null; | ||
|
||
do { | ||
const queries = [Query.limit(100)]; | ||
|
||
if (cursor) { | ||
queries.push(Query.cursorAfter(cursor)); | ||
} | ||
|
||
const { documents } = await databases.listDocuments( | ||
Bun.env.APPWRITE_DATABASE_ID, | ||
Bun.env.APPWRITE_COLLECTION_ID, | ||
queries | ||
); | ||
|
||
if (documents.length > 0) { | ||
cursor = documents[documents.length - 1].$id; | ||
} else { | ||
log(`No more documents found.`); | ||
cursor = null; | ||
break; | ||
} | ||
|
||
log(`Syncing chunk of ${documents.length} documents ...`); | ||
await index.addDocuments(documents, { primaryKey: '$id' }); | ||
} while (cursor !== null); | ||
|
||
log('Sync finished.'); | ||
|
||
return res.send('Sync finished.', 200); | ||
}; |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need these?
We want to do things the bun way