diff --git a/README.md b/README.md index 4d4071b..880bb4e 100644 --- a/README.md +++ b/README.md @@ -7,9 +7,10 @@ Instagram 게시물과 YouTube 영상의 텍스트를 레시피로 구조화해 - Google Authorization Code/OIDC 로그인, PKCE `S256`, 이메일 allowlist - HttpOnly 자체 인증 cookie와 그룹 멤버 간 recipe 공유 - Instagram caption 추출 및 YouTube 설명·자막·timestamp 추출 -- MiniMax OpenAI 호환 Chat Completions API를 이용한 레시피 구조화 +- 여러 URL을 영속 대기열에 등록하고 사용자가 요청할 때 서버에서 비동기 처리 +- 추출한 본문을 최대 3개·10만 자 단위로 묶어 MiniMax Chat Completions API로 레시피 구조화 - 영어 등 외국어 원문의 recipe 필드를 자연스러운 한국어로 번역 -- AI 결과 미리보기/편집 후 저장 +- AI 결과를 schema로 검증해 자동 저장하고 실패 작업은 대기열에서 재시도 - Recipe CRUD, 그룹 멤버 조회와 소유자·원본별 중복 방지 - 외부 이미지를 최대 1280px WebP로 변환해 `IMAGE_ROOT`에 저장 - 반응형 Recipe 목록/상세 화면과 YouTube timestamp 링크 @@ -59,7 +60,7 @@ MONGO_FALLBACK_URI mongodb://172.16.0.7:27017/our_recipe_atlas IMAGE_ROOT ./data/images ``` -개발과 운영 모두 `our_recipe_atlas` DB를 사용합니다. 개발 데이터는 `users_dev`, `recipes_dev` 컬렉션으로 분리되며, 로그인 허용 계정은 공통 `allowed_google_emails` 컬렉션을 사용합니다. 프로젝트를 `D:\project\our_recipe_atlas`에서 실행하면 이미지는 `D:\project\our_recipe_atlas\data\images\recipes\...`에 저장됩니다. +개발과 운영 모두 `our_recipe_atlas` DB를 사용합니다. 개발 데이터는 `users_dev`, `groups_dev`, `import_jobs_dev`, `recipes_dev` 컬렉션으로 분리되며, 로그인 허용 계정은 공통 `allowed_google_emails` 컬렉션을 사용합니다. 프로젝트를 `D:\project\our_recipe_atlas`에서 실행하면 이미지는 `D:\project\our_recipe_atlas\data\images\recipes\...`에 저장됩니다. Linux에서 자동 적용되는 운영 프로필: @@ -70,7 +71,7 @@ MONGO_FALLBACK_URI mongodb://172.16.0.7:27017/our_recipe_atlas IMAGE_ROOT /mnt/recipe-ssd/our_recipe_atlas/images ``` -운영 데이터는 같은 DB의 `users`, `recipes` 컬렉션에 저장되고 이미지는 `/mnt/recipe-ssd/our_recipe_atlas/images/recipes/...`에 저장됩니다. 이미지의 DB 값은 두 OS 모두 `recipes//cover.webp` 형식입니다. +운영 데이터는 같은 DB의 `users`, `groups`, `import_jobs`, `recipes` 컬렉션에 저장되고 이미지는 `/mnt/recipe-ssd/our_recipe_atlas/images/recipes/...`에 저장됩니다. 이미지의 DB 값은 두 OS 모두 `recipes//cover.webp` 형식입니다. Windows에서 만든 `node_modules`는 운영 서버로 복사하지 말고, `sharp` 등 OS별 바이너리가 Linux용으로 설치되도록 운영 서버에서 `npm ci --omit=dev`를 실행해야 합니다. @@ -138,7 +139,7 @@ MINIMAX_BASE_URL=https://api.minimax.io/v1 MINIMAX_MODEL=MiniMax-M2.7 ``` -파서는 최대 한 번만 repair 요청을 수행하며, 결과를 Zod schema로 검증한 뒤 미리보기로 반환합니다. 원문에 없는 재료와 수량을 추측하지 않도록 system prompt에 제한을 둡니다. +URL별로 본문을 먼저 추출한 뒤 최대 3개·원문 합계 10만 자씩 한 요청에 넣습니다. 배치 전체 또는 일부 결과가 유효하지 않으면 해당 원문만 개별 요청으로 재시도합니다. 각 결과는 Zod schema로 검증하며, 원문에 없는 재료와 수량을 추측하지 않도록 system prompt에 제한을 둡니다. ### Instagram @@ -159,10 +160,11 @@ ID와 비밀번호로 서버에서 자동 로그인하지 않습니다. cookie MongoDB는 `our_recipe_atlas` 하나만 사용하고 collection으로 환경을 분리합니다. - 공통: `allowed_google_emails` -- 개발: `users_dev`, `groups_dev`, `recipes_dev` -- 운영: `users`, `groups`, `recipes` +- 개발: `users_dev`, `groups_dev`, `import_jobs_dev`, `recipes_dev` +- 운영: `users`, `groups`, `import_jobs`, `recipes` - 각 `users*.googleSub`: unique index - 각 `groups*.memberGoogleSubs`: 조회 index +- 각 `import_jobs*`: 사용자·원본별 unique index와 상태 조회 index - 각 `recipes*`: `ownerGoogleSub + source.platform + source.sourceId` unique index - 원본 text와 YouTube transcript를 recipe source에 보존해 재분석에 사용할 수 있습니다. @@ -189,7 +191,11 @@ recipes//cover.webp | Method | Path | 역할 | | --- | --- | --- | -| `POST` | `/api/import/preview` | URL 추출 및 AI recipe 미리보기 | +| `POST` | `/api/import/jobs` | URL 목록을 사용자 대기열에 등록 | +| `GET` | `/api/import/jobs` | 대기·처리·실패 작업 조회 | +| `POST` | `/api/import/jobs/process` | 대기 작업의 서버 처리를 요청하고 즉시 `202` 반환 | +| `POST` | `/api/import/jobs/:id/retry` | 실패 작업을 대기 상태로 되돌림 | +| `POST` | `/api/import/preview` | 이전 클라이언트 호환용 단일 URL 미리보기 | | `GET` | `/api/recipes` | 내 그룹의 recipe 목록 | | `GET` | `/api/recipes/:id` | 내 그룹의 recipe 상세 | | `POST` | `/api/recipes` | 미리보기 확인 후 저장 | @@ -210,13 +216,13 @@ npm test npm run lint ``` -테스트 범위에는 URL/ID 판별, AI schema, timestamp 정규화, allowlist, 인증 middleware, 그룹 공유와 소유자 쓰기 권한, CRUD, 중복 source 처리, 이미지 상대경로와 SSRF 차단이 포함됩니다. +테스트 범위에는 URL/ID 판별, MiniMax 본문 배치와 AI schema, 대기열 상태 전이, timestamp 정규화, allowlist, 인증 middleware, 그룹 공유와 소유자 쓰기 권한, CRUD, 중복 source 처리, 이미지 상대경로와 SSRF 차단이 포함됩니다. 실제 계정과 네트워크가 준비된 뒤에는 별도 smoke test로 다음을 확인합니다. 1. Instagram Reel/Post의 긴 caption 추출 2. 실제 YouTube 레시피 영상의 description, transcript, timestamp 추출 -3. MiniMax 응답 품질과 미리보기 수정/저장 +3. 여러 본문의 MiniMax 응답 품질과 자동 저장 4. 서버 재시작 후 MongoDB recipe와 SSD 이미지 표시 ## 운영 배포 예시 diff --git a/public/css/app.css b/public/css/app.css index 5aab985..472d98b 100644 --- a/public/css/app.css +++ b/public/css/app.css @@ -135,11 +135,22 @@ img { display: block; max-width: 100%; } .import-panel { display: grid; grid-template-columns: minmax(180px, 0.7fr) 2fr; gap: 30px 54px; align-items: center; } .import-panel h2, .preview-panel h2, .library h2 { margin: 4px 0 0; font-family: var(--serif); font-size: clamp(2rem, 4vw, 3.4rem); font-weight: 400; } -.import-form { display: flex; gap: 10px; padding: 8px; border-radius: 999px; background: var(--white); } -.import-form input { flex: 1; min-width: 0; padding: 10px 18px; border: 0; outline: 0; color: var(--ink); background: transparent; } +.import-form { display: flex; gap: 10px; align-items: flex-end; padding: 8px; border-radius: 18px; background: var(--white); } +.import-form textarea { flex: 1; min-width: 0; min-height: 112px; padding: 12px 14px; resize: vertical; border: 0; outline: 0; color: var(--ink); background: transparent; font: inherit; line-height: 1.5; } .status { grid-column: 2; padding: 12px 18px; border-radius: 12px; background: rgba(255, 255, 255, 0.1); } .status.success { color: #d8efc9; } .status.error { color: #ffd0c4; } +.import-queue { grid-column: 2; padding: 18px; border-radius: 18px; color: var(--ink); background: var(--white); } +.import-queue-heading, .import-queue-heading > div, .import-job { display: flex; align-items: center; justify-content: space-between; gap: 12px; } +.import-queue-heading h3 { margin: 0; font-family: var(--serif); font-size: 1.45rem; font-weight: 400; } +.import-job-list { display: grid; gap: 8px; margin-top: 14px; } +.import-job { padding: 12px 14px; border: 1px solid var(--line); border-radius: 12px; } +.import-job-body { display: grid; min-width: 0; gap: 4px; } +.import-job-body a { overflow: hidden; color: var(--ink); font-size: 0.86rem; text-overflow: ellipsis; white-space: nowrap; } +.import-job-body small { color: #9a3c2c; } +.import-job-status { width: fit-content; padding: 3px 7px; border-radius: 999px; color: var(--green); background: rgba(63, 91, 69, 0.1); font-size: 0.68rem; font-weight: 700; } +.import-job-processing .import-job-status { color: #9b4a21; background: rgba(216, 115, 55, 0.13); } +.import-job-failed .import-job-status { color: #9a3c2c; background: rgba(154, 60, 44, 0.12); } .preview-panel { margin-top: 28px; color: var(--ink); background: var(--white); border: 1px solid var(--line); } .section-heading, .editor-section-heading, .editor-group-heading { display: flex; align-items: center; justify-content: space-between; gap: 20px; } @@ -282,7 +293,7 @@ img { display: block; max-width: 100%; } @media (max-width: 860px) { .import-panel, .preview-layout, .detail-hero, .detail-columns { grid-template-columns: 1fr; } - .status { grid-column: 1; } + .status, .import-queue { grid-column: 1; } .recipe-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); } .detail-intro { min-height: 380px; } } diff --git a/public/index.html b/public/index.html index 36cbbe8..fceb5cb 100644 --- a/public/index.html +++ b/public/index.html @@ -28,29 +28,28 @@

새 레시피 가져오기

- - - + + +
- - -
-

03

+

02

Recipe Library

diff --git a/public/js/app.js b/public/js/app.js index 72b92e3..0596107 100644 --- a/public/js/app.js +++ b/public/js/app.js @@ -9,8 +9,10 @@ const appBar = mountAppBar({ onManageAccess: openAccessManager }); const elements = { importForm: document.querySelector('#import-form'), importStatus: document.querySelector('#import-status'), - previewSection: document.querySelector('#preview-section'), - previewForm: document.querySelector('#preview-form'), + importQueue: document.querySelector('#import-queue'), + importJobList: document.querySelector('#import-job-list'), + importJobCount: document.querySelector('#import-job-count'), + processImportJobs: document.querySelector('#process-import-jobs'), recipeGrid: document.querySelector('#recipe-grid'), emptyLibrary: document.querySelector('#empty-library'), recipeCount: document.querySelector('#recipe-count'), @@ -30,7 +32,8 @@ const elements = { toast: document.querySelector('#toast'), }; -let previewData = null; +let importJobs = []; +let importJobPollTimer = null; let savedRecipes = []; let currentUser = null; let searchQuery = ''; @@ -41,6 +44,7 @@ const selectedTags = new Set(); const SEARCH_DEBOUNCE_MS = 180; const VISIBLE_TAG_LIMIT = 6; +const IMPORT_JOB_POLL_MS = 3000; function escapeHtml(value) { return String(value ?? '') @@ -181,6 +185,75 @@ async function loadRecipes() { renderRecipes(); } +function importJobLabel(status) { + if (status === 'pending') return '대기 중'; + if (status === 'processing') return '처리 중'; + return '처리 실패'; +} + +function importJobUrlLabel(value) { + try { + const url = new URL(value); + return `${url.hostname}${url.pathname}${url.search}`; + } catch { + return value; + } +} + +function renderImportJobs() { + const pendingCount = importJobs.filter(({ status }) => status === 'pending').length; + const processingCount = importJobs.filter(({ status }) => status === 'processing').length; + elements.importQueue.hidden = importJobs.length === 0; + elements.importJobCount.textContent = `${importJobs.length}개`; + elements.processImportJobs.disabled = pendingCount === 0 && processingCount === 0; + elements.processImportJobs.textContent = processingCount > 0 + ? `처리 재요청 (${processingCount})` + : `처리 요청${pendingCount ? ` (${pendingCount})` : ''}`; + elements.importJobList.innerHTML = importJobs.map((job) => ` +
+
+ ${importJobLabel(job.status)} + ${escapeHtml(importJobUrlLabel(job.url))} + ${job.error?.message ? `${escapeHtml(job.error.message)}` : ''} +
+ ${job.status === 'failed' + ? `` + : ''} +
+ `).join(''); +} + +function scheduleImportJobPoll() { + window.clearTimeout(importJobPollTimer); + importJobPollTimer = null; + if (!importJobs.some(({ status }) => status === 'processing')) return; + importJobPollTimer = window.setTimeout(async () => { + try { + await loadImportJobs(); + } catch (error) { + showToast(error.message); + scheduleImportJobPoll(); + } + }, IMPORT_JOB_POLL_MS); +} + +async function loadImportJobs() { + const previousActiveIds = new Set( + importJobs + .filter(({ status }) => status === 'pending' || status === 'processing') + .map(({ id }) => id), + ); + const { jobs } = await apiRequest('/api/import/jobs'); + importJobs = jobs; + renderImportJobs(); + + const visibleIds = new Set(jobs.map(({ id }) => id)); + if ([...previousActiveIds].some((id) => !visibleIds.has(id))) { + await loadRecipes(); + } + scheduleImportJobPoll(); +} + async function loadAllowedEmails() { const { emails } = await apiRequest('/api/allowed-emails'); elements.allowedEmailList.innerHTML = emails.map((email) => { @@ -216,166 +289,60 @@ async function openAccessManager() { } } -function ingredientGroupTemplate(group, groupIndex) { - return `
-
- - -
-
- ${group.items.map((item, itemIndex) => `
- - - -
`).join('')} -
- -
`; -} - -function renderPreview() { - const { recipe, imagePreviewUrl, source } = previewData; - const image = safeImageUrl(imagePreviewUrl); - elements.previewForm.innerHTML = ` -
-
- ${image ? `가져온 레시피 미리보기` : 'NO IMAGE'} - ${escapeHtml(source.platform)} -
-
- - - -
-
-
-

재료

-
${recipe.ingredientGroups.map(ingredientGroupTemplate).join('')}
-
-
-

조리 순서

-
${recipe.steps.map((step, index) => `
- ${index + 1} - - - -
`).join('')}
-
-
- - -
-
- -
`; - elements.previewSection.hidden = false; - elements.previewSection.scrollIntoView({ behavior: 'smooth', block: 'start' }); -} - -function readPreviewForm() { - const ingredientGroups = [...elements.previewForm.querySelectorAll('[data-ingredient-group]')].map((group) => ({ - name: group.querySelector('[data-group-name]').value.trim(), - items: [...group.querySelectorAll('[data-ingredient-item]')].map((item) => ({ - name: item.querySelector('[data-item-name]').value.trim(), - amount: item.querySelector('[data-item-amount]').value.trim() || null, - })), - })); - const steps = [...elements.previewForm.querySelectorAll('[data-step]')].map((step, index) => { - const timestamp = step.querySelector('[data-step-time]').value; - return { - order: index + 1, - text: step.querySelector('[data-step-text]').value.trim(), - timestampSec: timestamp === '' ? null : Number(timestamp), - }; - }); - - return { - title: document.querySelector('#recipe-title').value.trim(), - summary: document.querySelector('#recipe-summary').value.trim() || null, - servings: document.querySelector('#recipe-servings').value.trim() || null, - ingredientGroups, - steps, - tips: document.querySelector('#recipe-tips').value.split(/\r?\n/).map((item) => item.trim()).filter(Boolean), - tags: document.querySelector('#recipe-tags').value.split(',').map((item) => item.trim()).filter(Boolean), - }; -} - -function mutatePreview(action, button) { - previewData.recipe = readPreviewForm(); - const groupIndex = Number(button.dataset.group); - const itemIndex = Number(button.dataset.item); - const stepIndex = Number(button.dataset.stepIndex); - - if (action === 'add-group') previewData.recipe.ingredientGroups.push({ name: '재료', items: [] }); - if (action === 'remove-group') previewData.recipe.ingredientGroups.splice(groupIndex, 1); - if (action === 'add-item') previewData.recipe.ingredientGroups[groupIndex].items.push({ name: '', amount: null }); - if (action === 'remove-item') previewData.recipe.ingredientGroups[groupIndex].items.splice(itemIndex, 1); - if (action === 'add-step') previewData.recipe.steps.push({ order: previewData.recipe.steps.length + 1, text: '', timestampSec: null }); - if (action === 'remove-step') previewData.recipe.steps.splice(stepIndex, 1); - previewData.recipe.steps.forEach((step, index) => { step.order = index + 1; }); - renderPreview(); -} - elements.importForm.addEventListener('submit', async (event) => { event.preventDefault(); const button = elements.importForm.querySelector('button'); button.disabled = true; - const timers = [ - window.setTimeout(() => setStatus('콘텐츠 가져오는 중...'), 500), - window.setTimeout(() => setStatus('레시피 분석 중...'), 1600), - ]; - setStatus('링크 확인 중...'); + setStatus('URL을 대기열에 추가하는 중...'); try { - const url = new FormData(elements.importForm).get('url'); - previewData = await apiRequest('/api/import/preview', { + const urls = String(new FormData(elements.importForm).get('urls') ?? '') + .split(/\s+/) + .map((url) => url.trim()) + .filter(Boolean); + const result = await apiRequest('/api/import/jobs', { method: 'POST', - body: JSON.stringify({ url }), + body: JSON.stringify({ urls }), }); - timers.forEach(window.clearTimeout); - setStatus('분석 완료', 'success'); - renderPreview(); + elements.importForm.reset(); + const duplicateMessage = result.duplicateCount + ? `, 중복 ${result.duplicateCount}개 제외` + : ''; + setStatus(`${result.addedCount}개를 대기열에 추가했습니다${duplicateMessage}.`, 'success'); + importJobs = result.jobs; + renderImportJobs(); + scheduleImportJobPoll(); } catch (error) { - timers.forEach(window.clearTimeout); setStatus(error.message, 'error'); } finally { button.disabled = false; } }); -elements.previewForm.addEventListener('click', (event) => { - const button = event.target.closest('[data-action]'); - if (button) mutatePreview(button.dataset.action, button); -}); - -elements.previewForm.addEventListener('submit', async (event) => { - event.preventDefault(); - const button = elements.previewForm.querySelector('[type="submit"]'); - button.disabled = true; +elements.processImportJobs.addEventListener('click', async () => { + elements.processImportJobs.disabled = true; try { - const { recipe } = await apiRequest('/api/recipes', { - method: 'POST', - body: JSON.stringify({ - recipe: readPreviewForm(), - source: previewData.source, - imagePreviewUrl: previewData.imagePreviewUrl, - }), - }); - previewData = null; - elements.previewSection.hidden = true; - elements.importForm.reset(); - showToast(`${recipe.title} 레시피를 저장했습니다.`); - await loadRecipes(); + await apiRequest('/api/import/jobs/process', { method: 'POST' }); + await loadImportJobs(); } catch (error) { showToast(error.message); - } finally { - button.disabled = false; + renderImportJobs(); } }); -document.querySelector('#cancel-preview').addEventListener('click', () => { - previewData = null; - elements.previewSection.hidden = true; +elements.importJobList.addEventListener('click', async (event) => { + const button = event.target.closest('[data-retry-import-job]'); + if (!button) return; + button.disabled = true; + try { + await apiRequest(`/api/import/jobs/${encodeURIComponent(button.dataset.retryImportJob)}/retry`, { + method: 'POST', + }); + await loadImportJobs(); + } catch (error) { + showToast(error.message); + button.disabled = false; + } }); elements.tagFilterOptions.addEventListener('click', (event) => { @@ -484,6 +451,7 @@ bindScrollToTop(elements.scrollToTop); currentUser = await loadSession(); appBar.setUser(currentUser); await loadRecipes(); +await loadImportJobs(); if ( currentUser.canManageAccess && new URLSearchParams(window.location.search).get('manage-access') === '1' diff --git a/src/app.js b/src/app.js index ba994fc..2735146 100644 --- a/src/app.js +++ b/src/app.js @@ -9,6 +9,8 @@ import importRoutes from './routes/import.routes.js'; import recipeRoutes from './routes/recipe.routes.js'; import { createSourceExtractor } from './services/extractors/index.js'; import { ImageStorageService } from './services/image-storage.service.js'; +import { ImportJobProcessorService } from './services/import-job-processor.service.js'; +import { RecipeCreatorService } from './services/recipe-creator.service.js'; import { RecipeParserService } from './services/recipe-parser.service.js'; import { AppError } from './utils/errors.js'; @@ -56,11 +58,13 @@ export async function buildApp({ config = loadConfig(), dependencies = {} } = {} if ( dependencies.allowedEmailRepository && dependencies.groupRepository + && dependencies.importJobRepository && dependencies.recipeRepository && dependencies.userRepository ) { app.decorate('allowedEmailRepository', dependencies.allowedEmailRepository); app.decorate('groupRepository', dependencies.groupRepository); + app.decorate('importJobRepository', dependencies.importJobRepository); app.decorate('recipeRepository', dependencies.recipeRepository); app.decorate('userRepository', dependencies.userRepository); } else { @@ -84,6 +88,27 @@ export async function buildApp({ config = loadConfig(), dependencies = {} } = {} 'imageStorage', dependencies.imageStorage ?? new ImageStorageService({ root: config.imageRoot }), ); + app.decorate( + 'recipeCreator', + dependencies.recipeCreator ?? new RecipeCreatorService({ + recipeRepository: app.recipeRepository, + imageStorage: app.imageStorage, + model: config.minimax.model, + }), + ); + app.decorate( + 'importJobProcessor', + dependencies.importJobProcessor ?? new ImportJobProcessorService({ + importJobRepository: app.importJobRepository, + sourceExtractor: app.sourceExtractor, + recipeParser: app.recipeParser, + recipeCreator: app.recipeCreator, + logger: app.log, + }), + ); + app.addHook('onClose', async () => { + await app.importJobProcessor.close(); + }); await app.register(accessRoutes); await app.register(importRoutes); diff --git a/src/config/env.js b/src/config/env.js index 5d49690..fc3b8d1 100644 --- a/src/config/env.js +++ b/src/config/env.js @@ -38,6 +38,7 @@ export function mongoCollectionNames(nodeEnv) { return Object.freeze({ allowedEmails: 'allowed_google_emails', groups: `groups${suffix}`, + importJobs: `import_jobs${suffix}`, recipes: `recipes${suffix}`, users: `users${suffix}`, }); diff --git a/src/plugins/mongo.js b/src/plugins/mongo.js index 0a3ad51..4d0a72a 100644 --- a/src/plugins/mongo.js +++ b/src/plugins/mongo.js @@ -2,6 +2,7 @@ import { MongoClient } from 'mongodb'; import fastifyPlugin from 'fastify-plugin'; import { AllowedEmailRepository } from '../repositories/allowed-email.repository.js'; import { GroupRepository } from '../repositories/group.repository.js'; +import { ImportJobRepository } from '../repositories/import-job.repository.js'; import { RecipeRepository } from '../repositories/recipe.repository.js'; import { UserRepository } from '../repositories/user.repository.js'; @@ -51,12 +52,14 @@ async function mongoPlugin(fastify, { config }) { const collections = config.mongoCollections; const allowedEmailRepository = new AllowedEmailRepository(db, collections.allowedEmails); const groupRepository = new GroupRepository(db, collections.groups); + const importJobRepository = new ImportJobRepository(db, collections.importJobs); const recipeRepository = new RecipeRepository(db, collections.recipes); const userRepository = new UserRepository(db, collections.users); await Promise.all([ allowedEmailRepository.ensureIndexes(), groupRepository.ensureIndexes(), + importJobRepository.ensureIndexes(), recipeRepository.ensureIndexes(), userRepository.ensureIndexes(), ]); @@ -69,6 +72,7 @@ async function mongoPlugin(fastify, { config }) { fastify.decorate('db', db); fastify.decorate('allowedEmailRepository', allowedEmailRepository); fastify.decorate('groupRepository', groupRepository); + fastify.decorate('importJobRepository', importJobRepository); fastify.decorate('recipeRepository', recipeRepository); fastify.decorate('userRepository', userRepository); diff --git a/src/repositories/import-job.repository.js b/src/repositories/import-job.repository.js new file mode 100644 index 0000000..0fa7f2f --- /dev/null +++ b/src/repositories/import-job.repository.js @@ -0,0 +1,140 @@ +import { randomUUID } from 'node:crypto'; + +const VISIBLE_STATUSES = ['pending', 'processing', 'failed']; + +export class ImportJobRepository { + constructor(db, collectionName = 'import_jobs') { + this.collection = db.collection(collectionName); + } + + async ensureIndexes() { + await this.collection.createIndex( + { ownerGoogleSub: 1, platform: 1, sourceId: 1 }, + { unique: true }, + ); + await this.collection.createIndex({ ownerGoogleSub: 1, status: 1, createdAt: 1 }); + } + + async enqueue({ ownerGoogleSub, url, platform, sourceId }) { + const now = new Date(); + const document = { + _id: randomUUID(), + ownerGoogleSub, + url, + platform, + sourceId, + status: 'pending', + attempts: 0, + error: null, + recipeId: null, + createdAt: now, + updatedAt: now, + startedAt: null, + completedAt: null, + lockedAt: null, + }; + + try { + await this.collection.insertOne(document); + return { created: true, job: document }; + } catch (error) { + if (error?.code !== 11000) throw error; + const job = await this.collection.findOne({ ownerGoogleSub, platform, sourceId }); + return { created: false, job }; + } + } + + async listVisible(ownerGoogleSub) { + return this.collection + .find({ ownerGoogleSub, status: { $in: VISIBLE_STATUSES } }) + .sort({ createdAt: 1 }) + .toArray(); + } + + async recoverStale(ownerGoogleSub, staleBefore) { + await this.collection.updateMany( + { + ownerGoogleSub, + status: 'processing', + lockedAt: { $lt: staleBefore }, + }, + { + $set: { + status: 'pending', + updatedAt: new Date(), + startedAt: null, + lockedAt: null, + }, + }, + ); + } + + async claimNext(ownerGoogleSub) { + const now = new Date(); + return this.collection.findOneAndUpdate( + { ownerGoogleSub, status: 'pending' }, + { + $set: { + status: 'processing', + error: null, + startedAt: now, + lockedAt: now, + updatedAt: now, + }, + $inc: { attempts: 1 }, + }, + { + sort: { createdAt: 1 }, + returnDocument: 'after', + includeResultMetadata: false, + }, + ); + } + + async markCompleted(jobId, recipeId) { + const now = new Date(); + await this.collection.updateOne( + { _id: jobId }, + { + $set: { + status: 'completed', + recipeId, + error: null, + completedAt: now, + lockedAt: null, + updatedAt: now, + }, + }, + ); + } + + async markFailed(jobId, error) { + await this.collection.updateOne( + { _id: jobId }, + { + $set: { + status: 'failed', + error, + lockedAt: null, + updatedAt: new Date(), + }, + }, + ); + } + + async retry(ownerGoogleSub, jobId) { + return this.collection.findOneAndUpdate( + { _id: jobId, ownerGoogleSub, status: 'failed' }, + { + $set: { + status: 'pending', + error: null, + startedAt: null, + lockedAt: null, + updatedAt: new Date(), + }, + }, + { returnDocument: 'after', includeResultMetadata: false }, + ); + } +} diff --git a/src/routes/import.routes.js b/src/routes/import.routes.js index c72c8e9..49d48c8 100644 --- a/src/routes/import.routes.js +++ b/src/routes/import.routes.js @@ -1,4 +1,38 @@ -import { importPreviewRequestSchema, sourceSchema } from '../schemas/recipe.schema.js'; +import { z } from 'zod'; +import { + enqueueImportJobsSchema, + importPreviewRequestSchema, + sourceSchema, +} from '../schemas/recipe.schema.js'; +import { NotFoundError } from '../utils/errors.js'; +import { + canonicalSourceUrl, + detectPlatform, + extractInstagramShortcode, + extractYouTubeId, +} from '../utils/url.js'; + +function toImportIdentity(value) { + const url = canonicalSourceUrl(value); + const platform = detectPlatform(url); + const sourceId = platform === 'youtube' + ? extractYouTubeId(url) + : extractInstagramShortcode(url); + return { url, platform, sourceId }; +} + +function toPublicJob(job) { + return { + id: String(job._id), + url: job.url, + platform: job.platform, + status: job.status, + attempts: job.attempts, + error: job.error, + createdAt: job.createdAt, + updatedAt: job.updatedAt, + }; +} export default async function importRoutes(fastify) { fastify.post('/api/import/preview', { preHandler: fastify.authenticate }, async (request) => { @@ -12,5 +46,57 @@ export default async function importRoutes(fastify) { imagePreviewUrl: source.thumbnailUrl, }; }); -} + fastify.post('/api/import/jobs', { preHandler: fastify.authenticate }, async (request, reply) => { + const { urls } = enqueueImportJobsSchema.parse(request.body); + const identities = [...new Map(urls.map((url) => { + const identity = toImportIdentity(url); + return [`${identity.platform}:${identity.sourceId}`, identity]; + })).values()]; + + let addedCount = 0; + let duplicateCount = urls.length - identities.length; + for (const identity of identities) { + const recipe = await fastify.recipeRepository.findBySource( + request.user.sub, + identity.platform, + identity.sourceId, + ); + if (recipe) { + duplicateCount += 1; + continue; + } + + const result = await fastify.importJobRepository.enqueue({ + ownerGoogleSub: request.user.sub, + ...identity, + }); + if (result.created) addedCount += 1; + else duplicateCount += 1; + } + + const jobs = await fastify.importJobRepository.listVisible(request.user.sub); + return reply.code(201).send({ + addedCount, + duplicateCount, + jobs: jobs.map(toPublicJob), + }); + }); + + fastify.get('/api/import/jobs', { preHandler: fastify.authenticate }, async (request) => { + const jobs = await fastify.importJobRepository.listVisible(request.user.sub); + return { jobs: jobs.map(toPublicJob) }; + }); + + fastify.post('/api/import/jobs/process', { preHandler: fastify.authenticate }, async (request, reply) => { + fastify.importJobProcessor.start(request.user.sub); + return reply.code(202).send(); + }); + + fastify.post('/api/import/jobs/:id/retry', { preHandler: fastify.authenticate }, async (request) => { + const { id } = z.object({ id: z.string().trim().min(1) }).parse(request.params); + const job = await fastify.importJobRepository.retry(request.user.sub, id); + if (!job) throw new NotFoundError('재시도할 가져오기 작업을 찾을 수 없습니다.'); + return { job: toPublicJob(job) }; + }); +} diff --git a/src/routes/recipe.routes.js b/src/routes/recipe.routes.js index 0afe5b2..bb1b22a 100644 --- a/src/routes/recipe.routes.js +++ b/src/routes/recipe.routes.js @@ -1,33 +1,5 @@ -import { randomUUID } from 'node:crypto'; import { recipePatchSchema, saveRecipeRequestSchema } from '../schemas/recipe.schema.js'; -import { ConflictError, NotFoundError, ValidationError } from '../utils/errors.js'; -import { - detectPlatform, - extractInstagramShortcode, - extractYouTubeId, -} from '../utils/url.js'; - -function toStoredSource(source) { - const { sourceUrl, ...rest } = source; - return { ...rest, url: sourceUrl }; -} - -function isDuplicateKeyError(error) { - return error?.code === 11000; -} - -function verifiedSource(source) { - const platform = detectPlatform(source.sourceUrl); - const sourceId = platform === 'youtube' - ? extractYouTubeId(source.sourceUrl) - : platform === 'instagram' - ? extractInstagramShortcode(source.sourceUrl) - : null; - if (platform !== source.platform || sourceId !== source.sourceId) { - throw new ValidationError('미리보기 원본 정보가 URL과 일치하지 않습니다.'); - } - return source; -} +import { NotFoundError } from '../utils/errors.js'; async function visibleOwnerGoogleSubs(fastify, googleSub) { const groupMembers = await fastify.groupRepository.listMemberGoogleSubs(googleSub); @@ -52,42 +24,11 @@ export default async function recipeRoutes(fastify) { fastify.post('/api/recipes', { preHandler: fastify.authenticate }, async (request, reply) => { const input = saveRecipeRequestSchema.parse(request.body); - verifiedSource(input.source); - const ownerGoogleSub = request.user.sub; - const duplicate = await fastify.recipeRepository.findBySource( - ownerGoogleSub, - input.source.platform, - input.source.sourceId, - ); - if (duplicate) throw new ConflictError(); - - const recipeId = randomUUID(); - let imagePath = null; - try { - const imageUrl = input.imagePreviewUrl ?? input.source.thumbnailUrl; - if (imageUrl) imagePath = await fastify.imageStorage.saveFromUrl(recipeId, imageUrl); - - const now = new Date(); - const document = { - _id: recipeId, - ownerGoogleSub, - ...input.recipe, - source: toStoredSource(input.source), - imagePath, - ai: { - provider: 'minimax', - model: fastify.config.minimax.model, - }, - createdAt: now, - updatedAt: now, - }; - const recipe = await fastify.recipeRepository.create(document); - return reply.code(201).send({ recipe }); - } catch (error) { - if (imagePath) await fastify.imageStorage.removeRecipe(recipeId).catch(() => {}); - if (isDuplicateKeyError(error)) throw new ConflictError(); - throw error; - } + const recipe = await fastify.recipeCreator.create({ + ownerGoogleSub: request.user.sub, + ...input, + }); + return reply.code(201).send({ recipe }); }); fastify.patch('/api/recipes/:id', { preHandler: fastify.authenticate }, async (request) => { diff --git a/src/schemas/recipe.schema.js b/src/schemas/recipe.schema.js index 69b7d39..cc2f492 100644 --- a/src/schemas/recipe.schema.js +++ b/src/schemas/recipe.schema.js @@ -49,6 +49,12 @@ export const importPreviewRequestSchema = z.object({ url: z.url(), }); +export const enqueueImportJobsSchema = z.object({ + urls: z.array(z.url()) + .min(1, 'URL을 하나 이상 입력해 주세요.') + .max(20, 'URL은 한 번에 최대 20개까지 등록할 수 있습니다.'), +}); + export const saveRecipeRequestSchema = z.object({ recipe: recipeDraftSchema, source: sourceSchema, diff --git a/src/services/import-job-processor.service.js b/src/services/import-job-processor.service.js new file mode 100644 index 0000000..17b6e54 --- /dev/null +++ b/src/services/import-job-processor.service.js @@ -0,0 +1,154 @@ +import { sourceSchema } from '../schemas/recipe.schema.js'; +import { AppError } from '../utils/errors.js'; + +const MAX_JOBS_PER_RUN = 20; +const MAX_BATCH_ITEMS = 3; +const MAX_BATCH_CHARACTERS = 100_000; + +function publicError(error) { + if (!(error instanceof AppError)) { + return { + code: 'IMPORT_PROCESSING_FAILED', + message: '가져오기 작업을 처리하지 못했습니다.', + }; + } + return { + code: error.code, + message: error.message, + }; +} + +export function createImportBatches(items) { + const batches = []; + let current = []; + let currentCharacters = 0; + + for (const item of items) { + const characters = item.source.rawText.length; + if ( + current.length > 0 + && (current.length >= MAX_BATCH_ITEMS + || currentCharacters + characters > MAX_BATCH_CHARACTERS) + ) { + batches.push(current); + current = []; + currentCharacters = 0; + } + current.push(item); + currentCharacters += characters; + } + + if (current.length > 0) batches.push(current); + return batches; +} + +export class ImportJobProcessorService { + constructor({ + importJobRepository, + sourceExtractor, + recipeParser, + recipeCreator, + logger = console, + }) { + this.importJobRepository = importJobRepository; + this.sourceExtractor = sourceExtractor; + this.recipeParser = recipeParser; + this.recipeCreator = recipeCreator; + this.logger = logger; + this.activeRuns = new Map(); + } + + start(ownerGoogleSub) { + if (this.activeRuns.has(ownerGoogleSub)) return false; + + const run = Promise.resolve() + .then(() => this.process(ownerGoogleSub)) + .catch((error) => { + this.logger.error({ error, ownerGoogleSub }, '가져오기 대기열 처리 실패'); + }) + .finally(() => { + this.activeRuns.delete(ownerGoogleSub); + }); + this.activeRuns.set(ownerGoogleSub, run); + return true; + } + + async waitForIdle(ownerGoogleSub) { + await this.activeRuns.get(ownerGoogleSub); + } + + async close() { + await Promise.allSettled(this.activeRuns.values()); + } + + async process(ownerGoogleSub) { + // 새 처리 요청이 시작되면 이전 서버 실행에서 중단된 작업도 다시 가져온다. + await this.importJobRepository.recoverStale( + ownerGoogleSub, + new Date(), + ); + + while (true) { + const jobs = []; + for (let index = 0; index < MAX_JOBS_PER_RUN; index += 1) { + const job = await this.importJobRepository.claimNext(ownerGoogleSub); + if (!job) break; + jobs.push(job); + } + if (jobs.length === 0) break; + + const extracted = []; + for (const job of jobs) { + try { + const source = sourceSchema.parse(await this.sourceExtractor.extract(job.url)); + if (source.rawText.length > MAX_BATCH_CHARACTERS) { + throw new AppError('원문이 100,000자를 초과해 분석할 수 없습니다.', { + statusCode: 422, + code: 'SOURCE_TEXT_TOO_LONG', + }); + } + extracted.push({ jobId: String(job._id), job, source }); + } catch (error) { + this.logger.warn?.({ error, jobId: job._id }, '가져오기 원문 추출 실패'); + await this.importJobRepository.markFailed(job._id, publicError(error)); + } + } + + for (const batch of createImportBatches(extracted)) { + let batchResults; + try { + batchResults = await this.recipeParser.parseMany( + batch.map(({ jobId, source }) => ({ jobId, source })), + ); + } catch { + batchResults = batch.map(({ jobId }) => ({ + jobId, + error: new Error('AI 배치 처리에 실패했습니다.'), + })); + } + const resultByJobId = new Map(batchResults.map((result) => [result.jobId, result])); + + for (const item of batch) { + try { + const batchResult = resultByJobId.get(item.jobId); + const recipe = batchResult?.recipe ?? await this.recipeParser.parse(item.source); + const saved = await this.recipeCreator.create({ + ownerGoogleSub, + recipe, + source: item.source, + imagePreviewUrl: item.source.thumbnailUrl, + }); + await this.importJobRepository.markCompleted(item.job._id, saved._id); + } catch (error) { + if (error?.code === 'DUPLICATE_RECIPE') { + await this.importJobRepository.markCompleted(item.job._id, null); + } else { + this.logger.warn?.({ error, jobId: item.job._id }, '가져오기 레시피 저장 실패'); + await this.importJobRepository.markFailed(item.job._id, publicError(error)); + } + } + } + } + } + } +} diff --git a/src/services/recipe-creator.service.js b/src/services/recipe-creator.service.js new file mode 100644 index 0000000..f2da164 --- /dev/null +++ b/src/services/recipe-creator.service.js @@ -0,0 +1,74 @@ +import { randomUUID } from 'node:crypto'; +import { ConflictError, ValidationError } from '../utils/errors.js'; +import { + detectPlatform, + extractInstagramShortcode, + extractYouTubeId, +} from '../utils/url.js'; + +function toStoredSource(source) { + const { sourceUrl, ...rest } = source; + return { ...rest, url: sourceUrl }; +} + +function isDuplicateKeyError(error) { + return error?.code === 11000; +} + +function verifiedSource(source) { + const platform = detectPlatform(source.sourceUrl); + const sourceId = platform === 'youtube' + ? extractYouTubeId(source.sourceUrl) + : platform === 'instagram' + ? extractInstagramShortcode(source.sourceUrl) + : null; + if (platform !== source.platform || sourceId !== source.sourceId) { + throw new ValidationError('미리보기 원본 정보가 URL과 일치하지 않습니다.'); + } + return source; +} + +export class RecipeCreatorService { + constructor({ recipeRepository, imageStorage, model }) { + this.recipeRepository = recipeRepository; + this.imageStorage = imageStorage; + this.model = model; + } + + async create({ ownerGoogleSub, recipe, source, imagePreviewUrl }) { + verifiedSource(source); + const duplicate = await this.recipeRepository.findBySource( + ownerGoogleSub, + source.platform, + source.sourceId, + ); + if (duplicate) throw new ConflictError(); + + const recipeId = randomUUID(); + let imagePath = null; + try { + const imageUrl = imagePreviewUrl ?? source.thumbnailUrl; + if (imageUrl) imagePath = await this.imageStorage.saveFromUrl(recipeId, imageUrl); + + const now = new Date(); + const document = { + _id: recipeId, + ownerGoogleSub, + ...recipe, + source: toStoredSource(source), + imagePath, + ai: { + provider: 'minimax', + model: this.model, + }, + createdAt: now, + updatedAt: now, + }; + return await this.recipeRepository.create(document); + } catch (error) { + if (imagePath) await this.imageStorage.removeRecipe(recipeId).catch(() => {}); + if (isDuplicateKeyError(error)) throw new ConflictError(); + throw error; + } + } +} diff --git a/src/services/recipe-parser.service.js b/src/services/recipe-parser.service.js index ca257ff..cb13c7c 100644 --- a/src/services/recipe-parser.service.js +++ b/src/services/recipe-parser.service.js @@ -5,8 +5,7 @@ import { AppError } from '../utils/errors.js'; const STANDARD_TAGS = STANDARD_RECIPE_TAGS.join(', '); -const SYSTEM_PROMPT = `당신은 원문에서 레시피를 구조화하는 데이터 변환기입니다. -반드시 JSON 객체만 출력하세요. +const RECIPE_RULES = `당신은 원문에서 레시피를 구조화하는 데이터 변환기입니다. 원칙: - 원문에 없는 재료, 수량, 조리 순서를 추측하거나 추가하지 않습니다. @@ -21,10 +20,9 @@ const SYSTEM_PROMPT = `당신은 원문에서 레시피를 구조화하는 데 - 원문에 재료 그룹 이름이 없으면 "재료"를 사용합니다. ingredientGroups[].name은 null이 될 수 없습니다. - 이 작업은 정보 추출이므로 깊은 분석은 필요하지 않습니다. - tags는 다음 표준 태그에서만 최대 3개를 선택합니다: ${STANDARD_TAGS} -- 재료명, 요리 고유명사, 식사 시간대는 tags에 넣지 않습니다. +- 재료명, 요리 고유명사, 식사 시간대는 tags에 넣지 않습니다.`; -출력 형식: -{ +const RECIPE_OUTPUT_EXAMPLE = `{ "title": "string", "summary": "string|null", "servings": "string|null", @@ -34,6 +32,24 @@ const SYSTEM_PROMPT = `당신은 원문에서 레시피를 구조화하는 데 "tags": ["string"] }`; +const SINGLE_SYSTEM_PROMPT = `${RECIPE_RULES} + +반드시 다음 형식의 JSON 객체 하나만 출력하세요. +${RECIPE_OUTPUT_EXAMPLE}`; + +const BATCH_SYSTEM_PROMPT = `${RECIPE_RULES} + +입력에는 서로 독립적인 여러 원문과 jobId가 들어 있습니다. 원문끼리 정보를 섞지 말고 입력된 모든 jobId에 대해 결과를 하나씩 만드세요. +반드시 다음 형식의 JSON 객체 하나만 출력하세요. +{ + "results": [ + { + "jobId": "입력과 동일한 string", + "recipe": ${RECIPE_OUTPUT_EXAMPLE} + } + ] +}`; + export function normalizeModelJson(content) { if (typeof content !== 'string' || !content.trim()) { throw new Error('AI 응답이 비어 있습니다.'); @@ -74,43 +90,59 @@ function buildUserPrompt(source) { return lines.filter(Boolean).join('\n\n'); } +function buildBatchUserPrompt(entries) { + return JSON.stringify({ + sources: entries.map(({ jobId, source }) => ({ + jobId, + platform: source.platform, + title: source.title, + rawText: source.rawText, + })), + }); +} + export class RecipeParserService { constructor({ apiKey, baseUrl, model, client } = {}) { this.model = model; this.client = client ?? (apiKey ? new OpenAI({ apiKey, baseURL: baseUrl }) : null); } - async parse(source) { + assertConfigured() { if (!this.client) { throw new AppError('MiniMax API가 설정되지 않았습니다.', { statusCode: 503, code: 'MINIMAX_NOT_CONFIGURED', }); } + } + + async createCompletion(systemPrompt, userPrompt, maxCompletionTokens) { + const response = await this.client.chat.completions.create({ + model: this.model, + messages: [ + { role: 'system', content: systemPrompt }, + { role: 'user', content: userPrompt }, + ], + reasoning_split: true, + max_completion_tokens: maxCompletionTokens, + temperature: 0.2, + stream: false, + }); + return response.choices?.[0]?.message?.content ?? ''; + } + + async parse(source) { + this.assertConfigured(); let lastError; let previousOutput = null; for (let attempt = 0; attempt < 2; attempt += 1) { try { - const response = await this.client.chat.completions.create({ - model: this.model, - messages: [ - { role: 'system', content: SYSTEM_PROMPT }, - { - role: 'user', - content: attempt === 0 - ? buildUserPrompt(source) - : `${buildUserPrompt(source)}\n\n이전 출력은 유효한 JSON 스키마가 아니었습니다. 수정해서 JSON 객체만 다시 출력하세요.\n이전 출력:\n${previousOutput}`, - }, - ], - reasoning_split: true, - max_completion_tokens: 8192, - temperature: 0.2, - stream: false, - }); - - previousOutput = response.choices?.[0]?.message?.content ?? ''; + const userPrompt = attempt === 0 + ? buildUserPrompt(source) + : `${buildUserPrompt(source)}\n\n이전 출력은 유효한 JSON 스키마가 아니었습니다. 수정해서 JSON 객체만 다시 출력하세요.\n이전 출력:\n${previousOutput}`; + previousOutput = await this.createCompletion(SINGLE_SYSTEM_PROMPT, userPrompt, 8192); const parsed = JSON.parse(normalizeModelJson(previousOutput)); return recipeDraftSchema.parse(normalizeRecipePayload(parsed)); } catch (error) { @@ -124,4 +156,53 @@ export class RecipeParserService { cause: lastError, }); } + + async parseMany(entries) { + this.assertConfigured(); + if (entries.length === 0) return []; + + let lastError; + let previousOutput = null; + const basePrompt = buildBatchUserPrompt(entries); + + for (let attempt = 0; attempt < 2; attempt += 1) { + try { + const userPrompt = attempt === 0 + ? basePrompt + : `${basePrompt}\n\n이전 출력은 유효한 배치 JSON이 아니었습니다. 입력된 모든 jobId를 포함해 수정하세요.\n이전 출력:\n${previousOutput}`; + const outputLimit = Math.min(32768, Math.max(8192, entries.length * 8192)); + previousOutput = await this.createCompletion( + BATCH_SYSTEM_PROMPT, + userPrompt, + outputLimit, + ); + const parsed = JSON.parse(normalizeModelJson(previousOutput)); + if (!Array.isArray(parsed.results)) throw new Error('배치 results가 없습니다.'); + + const byJobId = new Map(parsed.results.map((result) => [result?.jobId, result])); + return entries.map(({ jobId }) => { + const result = byJobId.get(jobId); + if (!result?.recipe) { + return { jobId, error: new Error('AI 배치 응답에 recipe가 없습니다.') }; + } + try { + return { + jobId, + recipe: recipeDraftSchema.parse(normalizeRecipePayload(result.recipe)), + }; + } catch (error) { + return { jobId, error }; + } + }); + } catch (error) { + lastError = error; + } + } + + throw new AppError('AI 배치 응답을 처리하지 못했습니다.', { + statusCode: 502, + code: 'AI_BATCH_RESPONSE_INVALID', + cause: lastError, + }); + } } diff --git a/tests/app.test.js b/tests/app.test.js index 89fbffc..8500f13 100644 --- a/tests/app.test.js +++ b/tests/app.test.js @@ -7,20 +7,29 @@ import { createTestConfig, FakeAllowedEmailRepository, FakeGroupRepository, + FakeImportJobRepository, FakeRecipeRepository, fakeUserRepository, recipeDraft, source, } from './helpers/fakes.js'; -function createDependencies(recipeRepository, groupRepository = new FakeGroupRepository()) { +function createDependencies( + recipeRepository, + groupRepository = new FakeGroupRepository(), + importJobRepository = new FakeImportJobRepository(), +) { return { allowedEmailRepository: new FakeAllowedEmailRepository(), groupRepository, + importJobRepository, recipeRepository, userRepository: fakeUserRepository, sourceExtractor: { extract: async () => source }, - recipeParser: { parse: async () => recipeDraft }, + recipeParser: { + parse: async () => recipeDraft, + parseMany: async (entries) => entries.map(({ jobId }) => ({ jobId, recipe: recipeDraft })), + }, imageStorage: { saveFromUrl: async (id) => `recipes/${id}/cover.webp`, removeRecipe: async () => {}, @@ -74,6 +83,10 @@ test('관리자는 이메일만으로 Google 로그인 허용 계정을 관리 assert.match(pageResponse.body, /id="site-header"/); assert.match(pageResponse.body, /id="access-overlay"/); assert.match(pageResponse.body, /id="scroll-to-top"/); + assert.match(pageResponse.body, /id="source-urls"/); + assert.match(pageResponse.body, /id="import-queue"/); + assert.match(pageResponse.body, /id="process-import-jobs"/); + assert.doesNotMatch(pageResponse.body, /id="preview-section"/); const appBarResponse = await app.inject({ method: 'GET', url: '/js/app-bar.js' }); assert.equal(appBarResponse.statusCode, 200); @@ -339,6 +352,103 @@ test('import preview와 소유자별 Recipe CRUD가 이어진다', async (t) => assert.equal(deleteResponse.statusCode, 204); }); +test('여러 URL을 대기열에 넣고 비동기로 일괄 처리하며 완료 작업은 숨긴다', async (t) => { + const recipeRepository = new FakeRecipeRepository(); + const importJobRepository = new FakeImportJobRepository(); + const dependencies = createDependencies( + recipeRepository, + new FakeGroupRepository(), + importJobRepository, + ); + const app = await buildApp({ config: createTestConfig(), dependencies }); + t.after(() => app.close()); + const cookie = authCookie(app); + + const enqueueResponse = await app.inject({ + method: 'POST', + url: '/api/import/jobs', + headers: { cookie }, + payload: { urls: [source.sourceUrl, source.sourceUrl] }, + }); + assert.equal(enqueueResponse.statusCode, 201); + assert.equal(enqueueResponse.json().addedCount, 1); + assert.equal(enqueueResponse.json().duplicateCount, 1); + assert.equal(enqueueResponse.json().jobs[0].status, 'pending'); + + const processResponse = await app.inject({ + method: 'POST', + url: '/api/import/jobs/process', + headers: { cookie }, + }); + assert.equal(processResponse.statusCode, 202); + await app.importJobProcessor.waitForIdle('google-user-1'); + + const queueResponse = await app.inject({ + method: 'GET', + url: '/api/import/jobs', + headers: { cookie }, + }); + assert.deepEqual(queueResponse.json().jobs, []); + + const recipesResponse = await app.inject({ + method: 'GET', + url: '/api/recipes', + headers: { cookie }, + }); + assert.equal(recipesResponse.json().recipes.length, 1); + + const duplicateResponse = await app.inject({ + method: 'POST', + url: '/api/import/jobs', + headers: { cookie }, + payload: { urls: [source.sourceUrl] }, + }); + assert.equal(duplicateResponse.json().addedCount, 0); + assert.equal(duplicateResponse.json().duplicateCount, 1); +}); + +test('가져오기 실패 작업은 대기열에 남고 사용자가 재시도할 수 있다', async (t) => { + const dependencies = createDependencies(new FakeRecipeRepository()); + dependencies.sourceExtractor = { + async extract() { + throw new Error('원문 추출 실패'); + }, + }; + const app = await buildApp({ config: createTestConfig(), dependencies }); + t.after(() => app.close()); + const cookie = authCookie(app); + + await app.inject({ + method: 'POST', + url: '/api/import/jobs', + headers: { cookie }, + payload: { urls: [source.sourceUrl] }, + }); + await app.inject({ + method: 'POST', + url: '/api/import/jobs/process', + headers: { cookie }, + }); + await app.importJobProcessor.waitForIdle('google-user-1'); + + const failedResponse = await app.inject({ + method: 'GET', + url: '/api/import/jobs', + headers: { cookie }, + }); + const [failedJob] = failedResponse.json().jobs; + assert.equal(failedJob.status, 'failed'); + assert.equal(failedJob.error.message, '가져오기 작업을 처리하지 못했습니다.'); + + const retryResponse = await app.inject({ + method: 'POST', + url: `/api/import/jobs/${failedJob.id}/retry`, + headers: { cookie }, + }); + assert.equal(retryResponse.statusCode, 200); + assert.equal(retryResponse.json().job.status, 'pending'); +}); + test('같은 그룹 사용자는 레시피를 함께 조회하지만 작성자만 수정하고 삭제한다', async (t) => { const recipeRepository = new FakeRecipeRepository(); const groupRepository = new FakeGroupRepository([ diff --git a/tests/helpers/fakes.js b/tests/helpers/fakes.js index 76c9a3b..f9b952f 100644 --- a/tests/helpers/fakes.js +++ b/tests/helpers/fakes.js @@ -60,6 +60,107 @@ export class FakeRecipeRepository { } } +export class FakeImportJobRepository { + constructor() { + this.jobs = new Map(); + this.nextId = 1; + } + + async enqueue({ ownerGoogleSub, url, platform, sourceId }) { + const existing = [...this.jobs.values()].find((job) => ( + job.ownerGoogleSub === ownerGoogleSub + && job.platform === platform + && job.sourceId === sourceId + )); + if (existing) return { created: false, job: existing }; + + const now = new Date(); + const job = { + _id: `job-${this.nextId}`, + ownerGoogleSub, + url, + platform, + sourceId, + status: 'pending', + attempts: 0, + error: null, + recipeId: null, + createdAt: now, + updatedAt: now, + startedAt: null, + completedAt: null, + lockedAt: null, + }; + this.nextId += 1; + this.jobs.set(job._id, job); + return { created: true, job }; + } + + async listVisible(ownerGoogleSub) { + return [...this.jobs.values()].filter((job) => ( + job.ownerGoogleSub === ownerGoogleSub + && ['pending', 'processing', 'failed'].includes(job.status) + )); + } + + async recoverStale(ownerGoogleSub, staleBefore) { + for (const job of this.jobs.values()) { + if ( + job.ownerGoogleSub === ownerGoogleSub + && job.status === 'processing' + && job.lockedAt < staleBefore + ) { + Object.assign(job, { status: 'pending', startedAt: null, lockedAt: null }); + } + } + } + + async claimNext(ownerGoogleSub) { + const job = [...this.jobs.values()].find((candidate) => ( + candidate.ownerGoogleSub === ownerGoogleSub && candidate.status === 'pending' + )); + if (!job) return null; + const now = new Date(); + Object.assign(job, { + status: 'processing', + error: null, + attempts: job.attempts + 1, + startedAt: now, + lockedAt: now, + updatedAt: now, + }); + return job; + } + + async markCompleted(jobId, recipeId) { + const job = this.jobs.get(jobId); + Object.assign(job, { + status: 'completed', + recipeId, + error: null, + completedAt: new Date(), + lockedAt: null, + }); + } + + async markFailed(jobId, error) { + Object.assign(this.jobs.get(jobId), { status: 'failed', error, lockedAt: null }); + } + + async retry(ownerGoogleSub, jobId) { + const job = this.jobs.get(jobId); + if (job?.ownerGoogleSub !== ownerGoogleSub || job.status !== 'failed') return null; + Object.assign(job, { + status: 'pending', + error: null, + startedAt: null, + lockedAt: null, + updatedAt: new Date(), + }); + return job; + } +} + export class FakeGroupRepository { constructor(groups = []) { this.groups = groups; diff --git a/tests/import-job-processor.test.js b/tests/import-job-processor.test.js new file mode 100644 index 0000000..311d8e9 --- /dev/null +++ b/tests/import-job-processor.test.js @@ -0,0 +1,76 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { + createImportBatches, + ImportJobProcessorService, +} from '../src/services/import-job-processor.service.js'; +import { FakeImportJobRepository, recipeDraft, source } from './helpers/fakes.js'; + +function extractedItem(jobId, rawText) { + return { + jobId, + source: { rawText }, + }; +} + +test('가져오기 본문은 최대 3개와 합계 100,000자 단위로 묶는다', () => { + const batches = createImportBatches([ + extractedItem('job-1', 'a'.repeat(40_000)), + extractedItem('job-2', 'b'.repeat(40_000)), + extractedItem('job-3', 'c'.repeat(30_000)), + extractedItem('job-4', 'd'), + extractedItem('job-5', 'e'), + ]); + + assert.deepEqual(batches.map((batch) => batch.map(({ jobId }) => jobId)), [ + ['job-1', 'job-2'], + ['job-3', 'job-4', 'job-5'], + ]); +}); + +test('처리 요청 하나가 대기열 전체를 3개씩 MiniMax 배치 처리한다', async () => { + const importJobRepository = new FakeImportJobRepository(); + for (let index = 1; index <= 4; index += 1) { + await importJobRepository.enqueue({ + ownerGoogleSub: 'google-user-1', + url: `https://www.instagram.com/reel/Recipe${index}/`, + platform: 'instagram', + sourceId: `Recipe${index}`, + }); + } + + const batchJobIds = []; + const processor = new ImportJobProcessorService({ + importJobRepository, + sourceExtractor: { + async extract(url) { + const sourceId = new URL(url).pathname.split('/').filter(Boolean)[1]; + return { + ...source, + sourceUrl: url, + sourceId, + rawText: `${sourceId} 본문`, + }; + }, + }, + recipeParser: { + async parseMany(entries) { + batchJobIds.push(entries.map(({ jobId }) => jobId)); + return entries.map(({ jobId }) => ({ jobId, recipe: recipeDraft })); + }, + }, + recipeCreator: { + async create({ source: recipeSource }) { + return { _id: `recipe-${recipeSource.sourceId}` }; + }, + }, + logger: { error() {} }, + }); + + processor.start('google-user-1'); + await processor.waitForIdle('google-user-1'); + + assert.deepEqual(batchJobIds.map((ids) => ids.length), [3, 1]); + assert.deepEqual(await importJobRepository.listVisible('google-user-1'), []); + assert.ok([...importJobRepository.jobs.values()].every(({ status }) => status === 'completed')); +}); diff --git a/tests/mongo.test.js b/tests/mongo.test.js index d5cef58..46c3f3f 100644 --- a/tests/mongo.test.js +++ b/tests/mongo.test.js @@ -24,6 +24,7 @@ test('Windows에서 개발용 MongoDB와 로컬 이미지 프로필을 자동 assert.deepEqual(config.mongoCollections, { allowedEmails: 'allowed_google_emails', groups: 'groups_dev', + importJobs: 'import_jobs_dev', recipes: 'recipes_dev', users: 'users_dev', }); @@ -45,6 +46,7 @@ test('OS 프로필이 환경파일의 개발·운영 선택값보다 우선한 assert.deepEqual(linuxConfig.mongoCollections, { allowedEmails: 'allowed_google_emails', groups: 'groups', + importJobs: 'import_jobs', recipes: 'recipes', users: 'users', }); diff --git a/tests/schema.test.js b/tests/schema.test.js index 48fc631..cda86db 100644 --- a/tests/schema.test.js +++ b/tests/schema.test.js @@ -75,3 +75,36 @@ test('MiniMax 사고 과정 분리와 충분한 출력 한도를 요청한다', assert.match(request.messages[0].content, /수량, 온도, 시간 값은 변환하지 않습니다/); assert.equal(result.ingredientGroups[0].name, '재료'); }); + +test('여러 본문과 jobId를 MiniMax 단일 요청으로 보내고 각각 검증한다', async () => { + const requests = []; + const client = { + chat: { + completions: { + async create(parameters) { + requests.push(parameters); + const sources = JSON.parse(parameters.messages[1].content).sources; + return { + choices: [{ + message: { + content: JSON.stringify({ + results: sources.map(({ jobId }) => ({ jobId, recipe: recipeDraft })), + }), + }, + }], + }; + }, + }, + }, + }; + const service = new RecipeParserService({ client, model: 'MiniMax-M2.7' }); + const results = await service.parseMany([ + { jobId: 'job-1', source: { platform: 'instagram', title: null, rawText: '본문 1' } }, + { jobId: 'job-2', source: { platform: 'youtube', title: '제목', rawText: '본문 2' } }, + ]); + + assert.equal(requests.length, 1); + assert.equal(requests[0].max_completion_tokens, 16384); + assert.deepEqual(results.map(({ jobId }) => jobId), ['job-1', 'job-2']); + assert.deepEqual(results.map(({ recipe }) => recipe), [recipeDraft, recipeDraft]); +});