Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Sync with meilisearch in bun runtime #223

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 145 additions & 0 deletions bun/sync-with-meilisearch/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
### Node ###
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
lerna-debug.log*
.pnpm-debug.log*

# Diagnostic reports (https://nodejs.org/api/report.html)
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json

# Runtime data
pids
*.pid
*.seed
*.pid.lock

# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov

# Coverage directory used by tools like istanbul
coverage
*.lcov

# nyc test coverage
.nyc_output

# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
.grunt

# Bower dependency directory (https://bower.io/)
bower_components

# node-waf configuration
.lock-wscript

# Compiled binary addons (https://nodejs.org/api/addons.html)
build/Release

# Dependency directories
node_modules/
jspm_packages/

# Snowpack dependency directory (https://snowpack.dev/)
web_modules/

# TypeScript cache
*.tsbuildinfo

# Optional npm cache directory
.npm

# Optional eslint cache
.eslintcache

# Optional stylelint cache
.stylelintcache

# Microbundle cache
.rpt2_cache/
.rts2_cache_cjs/
.rts2_cache_es/
.rts2_cache_umd/

# Optional REPL history
.node_repl_history

# Output of 'npm pack'
*.tgz

# Yarn Integrity file
.yarn-integrity

# dotenv environment variable files
.env
.env.development.local
.env.test.local
.env.production.local
.env.local

# parcel-bundler cache (https://parceljs.org/)
.cache
.parcel-cache

# Next.js build output
.next
out

# Nuxt.js build / generate output
.nuxt
dist

# Gatsby files
.cache/
# Comment in the public line in if your project uses Gatsby and not Next.js
# https://nextjs.org/blog/next-9-1#public-directory-support
# public

# vuepress build output
.vuepress/dist

# vuepress v2.x temp and cache directory
.temp

# Docusaurus cache and generated files
.docusaurus

# Serverless directories
.serverless/

# FuseBox cache
.fusebox/

# DynamoDB Local files
.dynamodb/

# TernJS port file
.tern-port

# Stores VSCode versions used for testing VSCode extensions
.vscode-test

# yarn v2
.yarn/cache
.yarn/unplugged
.yarn/build-state.yml
.yarn/install-state.gz
.pnp.*

### Node Patch ###
# Serverless Webpack directories
.webpack/

# Optional stylelint cache

# SvelteKit build / generate output
.svelte-kit

# End of https://www.toptal.com/developers/gitignore/api/node

# OS
## Mac
.DS_Store
106 changes: 106 additions & 0 deletions bun/sync-with-meilisearch/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# ⚡ Bun Sync with Meilisearch Function

Syncs documents in an Appwrite database collection to a Meilisearch index.

## 🧰 Usage

### GET /

Returns HTML page where search can be performed to test the indexing.

### POST /

Triggers indexing of the Appwrite database collection to Meilisearch.

**Response**

Sample `204` Response: No content.

## ⚙️ Configuration

| Setting | Value |
| ----------------- | ------------- |
| Runtime | Bun (1.0) |
| Entrypoint | `src/main.ts` |
| Build Commands | `bun install` |
| Permissions | `any` |
| Timeout (Seconds) | 15 |

## 🔒 Environment Variables

### APPWRITE_API_KEY

API Key to talk to Appwrite backend APIs.

| Question | Answer |
| ------------- | -------------------------------------------------------------------------------------------------- |
| Required | Yes |
| Sample Value | `d1efb...aec35` |
| Documentation | [Appwrite: Getting Started for Server](https://appwrite.io/docs/getting-started-for-server#apiKey) |

### APPWRITE_DATABASE_ID

The ID of the Appwrite database that contains the collection to sync.

| Question | Answer |
| ------------- | --------------------------------------------------------- |
| Required | Yes |
| Sample Value | `612a3...5b6c9` |
| Documentation | [Appwrite: Databases](https://appwrite.io/docs/databases) |

### APPWRITE_COLLECTION_ID

The ID of the collection in the Appwrite database to sync.

| Question | Answer |
| ------------- | ------------------------------------------------------------- |
| Required | Yes |
| Sample Value | `7c3e8...2a9f1` |
| Documentation | [Appwrite: Collections](https://appwrite.io/docs/databases#collection) |

### APPWRITE_ENDPOINT

The URL endpoint of the Appwrite server. If not provided, it defaults to the Appwrite Cloud server: `https://cloud.appwrite.io/v1`.

| Question | Answer |
| ------------ | ------------------------------ |
| Required | No |
| Sample Value | `https://cloud.appwrite.io/v1` |

### MEILISEARCH_ENDPOINT

The host URL of the Meilisearch server.

| Question | Answer |
| ------------ | ----------------------- |
| Required | Yes |
| Sample Value | `http://127.0.0.1:7700` |

### MEILISEARCH_ADMIN_API_KEY

The admin API key for Meilisearch.

| Question | Answer |
| ------------- | ------------------------------------------------------------------------ |
| Required | Yes |
| Sample Value | `masterKey1234` |
| Documentation | [Meilisearch: API Keys](https://docs.meilisearch.com/reference/api/keys) |

### MEILISEARCH_INDEX_NAME

Name of the Meilisearch index to which the documents will be synchronized.

| Question | Answer |
| ------------ | ---------- |
| Required | Yes |
| Sample Value | `my_index` |

### MEILISEARCH_SEARCH_API_KEY

API Key for Meilisearch search operations.

| Question | Answer |
| ------------- | ------------------------------------------------------------------------ |
| Required | Yes |
| Sample Value | `searchKey1234` |
| Documentation | [Meilisearch: API Keys](https://docs.meilisearch.com/reference/api/keys) |
Binary file added bun/sync-with-meilisearch/bun.lockb
Binary file not shown.
15 changes: 15 additions & 0 deletions bun/sync-with-meilisearch/env.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
declare module "bun" {
interface Env {
APPWRITE_ENDPOINT?: string;
APPWRITE_API_KEY: string;
APPWRITE_FUNCTION_PROJECT_ID: string;
APPWRITE_DATABASE_ID: string;
APPWRITE_COLLECTION_ID: string;
MEILISEARCH_ENDPOINT: string;
MEILISEARCH_INDEX_NAME: string;
MEILISEARCH_ADMIN_API_KEY: string;
MEILISEARCH_SEARCH_API_KEY: string;
}
}

export { };
20 changes: 20 additions & 0 deletions bun/sync-with-meilisearch/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "sync-with-meilisearch",
"version": "1.0.0",
"description": "",
"main": "src/main.ts",
"type": "module",
"scripts": {
"format": "prettier --write ."
},
"dependencies": {
"bun-types": "^1.0.7",
"meilisearch": "^0.35.0",
"node-appwrite": "^11.0.0",
"path": "^0.12.7",
"url": "^0.11.3"
Comment on lines +14 to +15
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need these?
We want to do things the bun way

},
"devDependencies": {
"prettier": "^3.0.0"
}
}
72 changes: 72 additions & 0 deletions bun/sync-with-meilisearch/src/main.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
import { Client, Databases, Query } from 'node-appwrite';
import { getStaticFile, interpolate, throwIfMissing } from './utils';
import { MeiliSearch } from 'meilisearch';

export default async ({ req, res, log }) => {
throwIfMissing(Bun.env, [
'APPWRITE_API_KEY',
'APPWRITE_DATABASE_ID',
'APPWRITE_COLLECTION_ID',
'MEILISEARCH_ENDPOINT',
'MEILISEARCH_INDEX_NAME',
'MEILISEARCH_ADMIN_API_KEY',
'MEILISEARCH_SEARCH_API_KEY',
]);

if (req.method === 'GET') {
const html = interpolate(await getStaticFile('index.html'), {
MEILISEARCH_ENDPOINT: Bun.env.MEILISEARCH_ENDPOINT,
MEILISEARCH_INDEX_NAME: Bun.env.MEILISEARCH_INDEX_NAME,
MEILISEARCH_SEARCH_API_KEY: Bun.env.MEILISEARCH_SEARCH_API_KEY,
});

return res.send(html, 200, { 'Content-Type': 'text/html; charset=utf-8' });
}

const client = new Client()
.setEndpoint(
Bun.env.APPWRITE_ENDPOINT ?? 'https://cloud.appwrite.io/v1'
)
.setProject(Bun.env.APPWRITE_FUNCTION_PROJECT_ID)
.setKey(Bun.env.APPWRITE_API_KEY);

const databases = new Databases(client);

const meilisearch = new MeiliSearch({
host: Bun.env.MEILISEARCH_ENDPOINT,
apiKey: Bun.env.MEILISEARCH_ADMIN_API_KEY,
});

const index = meilisearch.index(Bun.env.MEILISEARCH_INDEX_NAME);

let cursor = null;

do {
const queries = [Query.limit(100)];

if (cursor) {
queries.push(Query.cursorAfter(cursor));
}

const { documents } = await databases.listDocuments(
Bun.env.APPWRITE_DATABASE_ID,
Bun.env.APPWRITE_COLLECTION_ID,
queries
);

if (documents.length > 0) {
cursor = documents[documents.length - 1].$id;
} else {
log(`No more documents found.`);
cursor = null;
break;
}

log(`Syncing chunk of ${documents.length} documents ...`);
await index.addDocuments(documents, { primaryKey: '$id' });
} while (cursor !== null);

log('Sync finished.');

return res.send('Sync finished.', 200);
};
Loading