Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- updateDataLog 및 log/ 폴더 관련 코드 전체 제거 (DB가 성공/실패 상태 관리) - saveCrawlYoutubeFailedKYSongs 제거 (crawlYoutube가 DB 방식으로 전환되어 미사용) - replaceSupabaseFailed에서 loadValidKYSongs → loadCrawlYoutubeFailedKYSongs 버그 수정 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Refactor/crawling
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fix/auto complete
fix : yml 액션 파일 환경 변수 받게 수정
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Chore/ky verify action
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix : crawlYoutubeVerify 우분투 환경 Puppeteer 샌드박스 옵션 추가
fix : 레이아웃 간격 조정
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Review Summary by QodoRefactor KY validation to AI-based matching and remove file-based logging
WalkthroughsDescription• Replace KY song validation with AI-based matching using GPT-4o-mini • Refactor crawling checkpoint system from file-based to database-backed • Remove unused query package and file-based logging infrastructure • Improve autocomplete label matching to require exact match before substitution • Update artist alias and Korean-Japanese mapping data Diagramflowchart LR
A["KY Crawling<br/>crawlYoutube.ts"] -->|YouTube Search| B["Extract Number"]
B -->|Validate| C["isValidKYExistNumber"]
C -->|AI Match| D["validateSongMatch<br/>GPT-4o-mini"]
D -->|Valid| E["postVerifyKySongsDB<br/>verify_ky_songs table"]
D -->|Invalid| F["postInvalidKYSongsDB<br/>invalid_ky_songs table"]
E -->|Checkpoint| G["getVerifyKySongsDB<br/>Resume Support"]
H["Old System<br/>File-based Logs"] -.->|Removed| I["New System<br/>DB Checkpoints"]
File Changes1. apps/web/src/hooks/useSearchSong.ts
|
Code Review by Qodo
1. ky-update premature exit
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
| "ky-valid": "tsx src/crawling/crawlYoutubeValid.ts", | ||
| "ky-update": "pnpm run ky-youtube & pnpm run ky-valid", | ||
| "ky-verify": "tsx src/crawling/crawlYoutubeVerify.ts", | ||
| "ky-update": "pnpm run ky-youtube & pnpm run ky-verify", |
There was a problem hiding this comment.
1. Ky-update premature exit 🐞 Bug ⛯ Reliability
packages/crawling의 ky-update가 &로 ky-youtube를 백그라운드 실행하고 wait하지 않아, ky-verify 종료 시점에 전체 스크립트가 끝나면서 ky-youtube가 중간에 종료되거나 결과가 불완전해질 수 있습니다.
Agent Prompt
### Issue description
`ky-update` 스크립트가 `&`로 백그라운드 실행을 만들고 `wait` 없이 종료되어, 한 작업이 끝나기 전에 스크립트가 종료되거나 실패가 전파되지 않습니다.
### Issue Context
현재 `ky-update`는 ky-youtube(백그라운드) + ky-verify(포그라운드) 형태입니다.
### Fix Focus Areas
- packages/crawling/package.json[9-19]
### Suggested implementation
- 옵션 A: `pnpm run ky-youtube & pnpm run ky-verify; wait`
- 옵션 B(권장): `concurrently -k -s first "pnpm run ky-youtube" "pnpm run ky-verify"` 추가 및 exit code 전파
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+13
to
+50
| const browser = await puppeteer.launch({ | ||
| headless: true, | ||
| args: ['--no-sandbox', '--disable-setuid-sandbox'], | ||
| }); | ||
| const page = await browser.newPage(); | ||
|
|
||
| const data = await getSongsKyNotNullDB(); | ||
| const verifiedIds = await getVerifyKySongsDB(); | ||
|
|
||
| console.log('getSongsKyNotNullDB : ', data.length); | ||
| console.log('이미 검증된 곡 수 : ', verifiedIds.size); | ||
| let index = 0; | ||
|
|
||
| for (const song of data) { | ||
| if (verifiedIds.has(song.id!)) { | ||
| continue; | ||
| } | ||
|
|
||
| console.log(song.title, ' - ', song.artist + ' : ', song.num_ky); | ||
| let isValid = true; | ||
| try { | ||
| isValid = await isValidKYExistNumber(page, song.num_ky, song.title, song.artist); | ||
| } catch (error) { | ||
| index++; | ||
| continue; | ||
| } | ||
|
|
||
| if (isValid) { | ||
| await postVerifyKySongsDB(song); | ||
| } else { | ||
| await updateSongsKyDB({ ...song, num_ky: null }); | ||
| } | ||
|
|
||
| index++; | ||
| console.log('crawlYoutubeVerify : ', index); | ||
| } | ||
|
|
||
| browser.close(); |
There was a problem hiding this comment.
2. Puppeteer leak/hang risk 🐞 Bug ⛯ Reliability
crawlYoutubeVerify가 Puppeteer를 top-level에서 실행하고 try/finally 없이 browser.close()를 await하지 않아, 중간 예외 발생 시 브라우저 프로세스가 남거나 스크립트가 종료/행(hang)될 수 있습니다.
Agent Prompt
### Issue description
`crawlYoutubeVerify.ts`가 Puppeteer lifecycle을 보장하지 않고 `browser.close()`도 await하지 않아, 예외/중단 시 브라우저 프로세스가 남아 스크립트가 hang되거나 리소스가 누수될 수 있습니다.
### Issue Context
현재 verify 스크립트는 top-level await로 실행되며, 종료 정리 로직이 보장되지 않습니다.
### Fix Focus Areas
- packages/crawling/src/crawling/crawlYoutubeVerify.ts[13-50]
### Suggested implementation
- `const main = async () => { ... } ; main();`
- `try { ... } catch { ... } finally { await browser.close(); }`
- `browser.close()`는 반드시 `await` 처리
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+75
to
+88
| export async function postVerifyKySongsDB(song: Song) { | ||
| const supabase = getClient(); | ||
|
|
||
| try { | ||
| const { id, title, artist } = song; | ||
| const { error } = await supabase.from('verify_ky_songs').insert({ id, title, artist }).select(); | ||
| if (error) { | ||
| console.error('postVerifyKySongsDB error : ', error); | ||
| } | ||
| return true; | ||
| } catch (error) { | ||
| console.error('catch - postVerifyKySongsDB error : ', error); | ||
| return error; | ||
| } |
There was a problem hiding this comment.
3. Verify insert errors hidden 🐞 Bug ⛯ Reliability
postVerifyKySongsDB가 Supabase insert error가 발생해도 true를 반환해 호출자가 실패를 감지할 수 없고, 검증 마킹 누락이 발생해도 다음 실행에서 반복 처리될 수 있습니다.
Agent Prompt
### Issue description
DB insert가 실패해도 `postVerifyKySongsDB`가 성공(true)으로 처리되어 실패 감지/대응이 불가능합니다. 또한 catch에서 error 객체를 반환해 반환 타입이 불일치합니다.
### Issue Context
verify 파이프라인의 체크포인트 테이블(verify_ky_songs)에 대한 기록이 실패해도 상위 로직은 알 수 없습니다.
### Fix Focus Areas
- packages/crawling/src/supabase/postDB.ts[75-89]
### Suggested implementation
- 함수 시그니처를 `Promise<boolean>`로 명확화
- `if (error) { console.error(...); return false; }`
- 또는 `if (error) throw error;`로 변경 후 호출부에서 try/catch 처리
- `.select()` 결과를 사용하지 않으면 제거
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📌 PR 제목
[Type] : 작업 내용 요약
📌 변경 사항
💬 추가 참고 사항