Commit Graph

84 Commits

Author SHA1 Message Date
Igor Loskutov
009590c080 feat: search frontend (#551)
* feat: better highlight

* feat(search): add long_summary to search vector for improved search results

- Update search vector to include long_summary with weight B (between title A and webvtt C)
- Modify SearchController to fetch long_summary and prioritize its snippets
- Generate snippets from long_summary first (max 2), then from webvtt for remaining slots
- Add comprehensive tests for long_summary search functionality
- Create migration to update search_vector_en column in PostgreSQL

This improves search quality by including summarized content which often contains
key topics and themes that may not be explicitly mentioned in the transcript.

* fix: address code review feedback for search enhancements

- Fix test file inconsistencies by removing references to non-existent model fields
  - Comment out tests for unimplemented features (room_ids, status filters, date ranges)
  - Update tests to only use currently available fields (room_id singular, no room_name/processing_status)
  - Mark future functionality tests with @pytest.mark.skip

- Make snippet counts configurable
  - Add LONG_SUMMARY_MAX_SNIPPETS constant (default: 2)
  - Replace hardcoded value with configurable constant

- Improve error handling consistency in WebVTT parsing
  - Use different log levels for different error types (debug for malformed, warning for decode, error for unexpected)
  - Add catch-all exception handler for unexpected errors
  - Include stack trace for critical errors

All existing tests pass with these changes.

* fix: correct datetime test to include required duration field

* feat: better highlight

* feat: search room names

* feat: acknowledge deleted room

* feat: search filters fix and rank removal

* chore: minor refactoring

* feat: better matches frontend

* chore: self-review (vibe)

* chore: self-review WIP

* chore: self-review WIP

* chore: self-review WIP

* chore: self-review WIP

* chore: self-review WIP

* chore: self-review WIP

* chore: self-review WIP

* remove swc (vibe)

* search url query sync (vibe)

* search url query sync (vibe)

* better casts and cap while

* PR review + simplify frontend hook

* pr: remove search db timeouts

* cleanup tests

* tests cleanup

* frontend cleanup

* index declarations

* refactor frontend (self-review)

* fix search pagination

* clear "x" for search input

* pagination max pages fix

* chore: cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup

* lockfile

* pr review
2025-08-20 20:56:45 -04:00
9eab952c63 feat: postgresql migration and removal of sqlite in pytest (#546)
* feat: remove support of sqlite, 100% postgres

* fix: more migration and make datetime timezone aware in postgres

* fix: change how database is get, and use contextvar to have difference instance between different loops

* test: properly use client fixture that handle lifetime/database connection

* fix: add missing client fixture parameters to test functions

This commit fixes NameError issues where test functions were trying to use
the 'client' fixture but didn't have it as a parameter. The changes include:

1. Added 'client' parameter to test functions in:
   - test_transcripts_audio_download.py (6 functions including fixture)
   - test_transcripts_speaker.py (3 functions)
   - test_transcripts_upload.py (1 function)
   - test_transcripts_rtc_ws.py (2 functions + appserver fixture)

2. Resolved naming conflicts in test_transcripts_rtc_ws.py where both HTTP
   client and StreamClient were using variable name 'client'. StreamClient
   instances are now named 'stream_client' to avoid conflicts.

3. Added missing 'from reflector.app import app' import in rtc_ws tests.

Background: Previously implemented contextvars solution with get_database()
function resolves asyncio event loop conflicts in Celery tasks. The global
client fixture was also created to replace manual AsyncClient instances,
ensuring proper FastAPI application lifecycle management and database
connections during tests.

All tests now pass except for 2 pre-existing RTC WebSocket test failures
related to asyncpg connection issues unrelated to these fixes.

* fix: ensure task are correctly closed

* fix: make separate event loop for the live server

* fix: make default settings pointing at postgres

* build: remove pytest-docker deps out of dev, just tests group
2025-08-14 11:40:52 -06:00
Igor Loskutov
6fb5cb21c2 feat: search backend (#537)
* docs: transient docs

* chore: cleanup

* webvtt WIP

* webvtt field

* chore: webvtt tests comments

* chore: remove useless tests

* feat: search TASK.md

* feat: full text search by title/webvtt

* chore: search api task

* feat: search api

* feat: search API

* chore: rm task md

* chore: roll back unnecessary validators

* chore: pr review WIP

* chore: pr review WIP

* chore: pr review

* chore: top imports

* feat: better lint + ci

* feat: better lint + ci

* feat: better lint + ci

* feat: better lint + ci

* chore: lint

* chore: lint

* fix: db datetime definitions

* fix: flush() params

* fix: update transcript mutability expectation / test

* fix: update transcript mutability expectation / test

* chore: auto review

* chore: new controller extraction

* chore: new controller extraction

* chore: cleanup

* chore: review WIP

* chore: pr WIP

* chore: remove ci lint

* chore: openapi regeneration

* chore: openapi regeneration

* chore: postgres test doc

* fix: .dockerignore for arm binaries

* fix: .dockerignore for arm binaries

* fix: cap test loops

* fix: cap test loops

* fix: cap test loops

* fix: get_transcript_topics

* chore: remove flow.md docs and claude guidance

* chore: remove claude.md db doc

* chore: remove claude.md db doc

* chore: remove claude.md db doc

* chore: remove claude.md db doc
2025-08-13 10:03:38 -04:00
f5b82d44e3 style: use ruff for linting and formatting (#524) 2025-07-31 17:57:43 -06:00
2a2af5fff2 fix: remove fief out of the source code (#502)
* fix: remove fief out of the source code

* fix: remove corresponding test about migration
2025-07-21 21:09:05 -06:00
901a239952 feat: migrate from chakra 2 to chakra 3 (#500)
* feat: separate page into different component, greatly improving the loading and reactivity

* fix: various fixes

* feat: migrate to Chakra UI v3 - update theme, fix deprecated props

- Add whiteAlpha color palette with semantic tokens
- Update button recipe with fontWeight 600 and hover states
- Move Poppins font from theme to HTML tag className
- Fix deprecated props: isDisabled→disabled, align→alignItems/textAlign
- Remove button.css as styles are now handled by Chakra v3

* fix: complete Chakra UI v3 deprecated prop migrations

- Replace all isDisabled with disabled
- Replace all isChecked with checked
- Replace all isLoading with loading
- Replace all isOpen with open
- Replace all noOfLines with lineClamp
- Replace all align with alignItems on Flex/Stack components
- Replace all justify with justifyContent on Flex/Stack components
- Update temporary Select components to use new prop names
- Update REFACTOR2.md with completion status

* fix: add value prop to Menu.Item for proper hover states in Chakra v3

* fix: update browse page components for Chakra UI v3 compatibility

- Fix FilterSidebar status filter styling and prop usage
- Update Pagination component to use new Chakra v3 props and structure
- Refactor TranscriptTable to use modern Chakra patterns
- Clean up browse page layout and props
- Remove unused import from transcripts API view
- Enhance theme with additional semantic color tokens

* fix: polish browse page UI for Chakra v3

- Add rounded corners to FilterSidebar
- Adjust responsive breakpoints from md to lg for table/card view
- Add consistent font weights to table headers
- Improve card view typography and spacing
- Fix padding and margins for better mobile experience
- Remove unused table recipe from theme

* fix: padding

* fix: rework transcript page

* fix: more tidy layout for topic

* fix: share and privacy using chakra3 select

* fix: fix share and privacy select, now working, with closing dialog

* fix: complete Chakra UI v3 migration for share components and fix all TypeScript errors

- Refactor shareZulip.tsx to integrate modal content directly
- Replace react-select-search with Chakra UI v3 Select components using collection pattern
- Convert all Checkbox components to use v3 composable structure (Checkbox.Root, etc.)
- Fix Card components to use Card.Root and Card.Body
- Replace deprecated textColor prop with color prop
- Update Menu components to use v3 namespace pattern (Menu.Root, Menu.Trigger, etc.)
- Remove unused AlertDialog imports
- Fix useDisclosure hook changes (isOpen -> open)
- Replace UnorderedList with List.Root and ListItem with List.Item
- Fix Skeleton components by removing isLoaded prop and using conditional rendering
- Update Button variants to valid v3 options
- Fix Spinner props (remove thickness, speed, emptyColor)
- Update toast API to use custom toaster component
- Fix Progress components and FormControl to Field.Root
- Update Alert to use compound component pattern
- Remove shareModal.tsx file after integration

* fix: bring back topic list

* fix: normalize menu item

* fix: migrate rooms page to Chakra UI v3 pattern

- Updated layout to match browse page with Flex container and proper spacing
- Migrated add/edit room modal from custom HTML to Chakra UI v3 Dialog component
- Replaced all Select components with Chakra UI v3 Select using createListCollection
- Replaced FormControl/FormLabel/FormHelperText with Field.Root/Field.Label/Field.HelperText
- Removed inline styles and used Chakra props (mr={2} instead of style={{ marginRight: "8px" }})
- Fixed TypeScript interfaces removing OptionBase extension
- Fixed theme.ts accordion anatomy import issue

* refactor: convert rooms list to table view with responsive design

- Create RoomTable component for desktop view showing room details in columns
- Create RoomCards component for mobile/tablet responsive view
- Refactor RoomList to use table/card components based on screen size
- Display Zulip configuration, room size, and recording settings in table
- Remove unused RoomItem component
- Import Room type from API for proper typing

* refactor: extract RoomActionsMenu component to eliminate duplication

- Create RoomActionsMenu component for consistent room action menus
- Update RoomCards and RoomTable to use the new shared component
- Remove duplicated menu code from both components

* feat: add icons to TranscriptActionsMenu for consistency

- Add FaTrash icon for Delete action with red color
- Add FaArrowsRotate icon for Reprocess action
- Matches the pattern established in RoomActionsMenu

* refactor: update icons from Font Awesome to Lucide React

- Replace FaEllipsisVertical with LuMenu in menu triggers
- Replace FaLink with LuLink for copy URL buttons
- Replace FaPencil with LuPen for edit actions
- Replace FaTrash with LuTrash for delete actions
- Replace FaArrowsRotate with LuRotateCw for reprocess action
- Consistent icon library usage across all components

* refactor: little pass on the icons

* fix: lu icon

* fix: primary for button

* fix: recording page with mic selection

* fix: also fix duration

* fix: use combobox for share zulip

* fix: use proper theming for button, variant was not recognized

* fix: room actions menu

* fix: remove other variant primary left.
2025-07-21 16:16:12 -06:00
fc38345d65 fix: separate browsing page into different components, limit to 10 by default (#498)
* feat: limit the amount of transcripts to 10 by default

* feat: separate page into different component, greatly improving the
loading and reactivity

* fix: current implementation immediately invokes the onDelete and
onReprocess

From pr-agent-monadical: Suggestion: The current implementation
immediately invokes the onDelete and onReprocess functions when the
component renders, rather than when the menu items are clicked. This can
cause unexpected behavior and potential memory leaks. Use callback
functions that only execute when the menu items are actually clicked.
[possible issue, importance: 9]
2025-07-17 20:18:00 -06:00
baf2822b81 fix: browse page timestamps show UTC instead of user local time (#482)
* fix: browse page timestamps show UTC instead of user local time

Closes #474

* fix: tests
2025-07-15 21:17:53 -06:00
9deb717e5b refactor: improve transcript list performance (#480)
* refactor: improve transcript list performance

* fix: sync openapi

* fix: frontend types

* fix: remove drop table _alembic_tmp_meeting

* fix: remove create table too

* fix: remove uq_recording_object_key
2025-07-15 15:10:05 -06:00
Igor Loskutov
c23e0e07ef update audio-deleted flow 2025-06-18 15:43:50 -04:00
ac81617090 Allow shared transcripts deletion 2025-03-03 15:25:00 +01:00
670852a8b2 Replace requests with httpx 2025-02-10 17:29:53 +01:00
bc33c6fb88 Fix validation 2024-10-04 18:24:45 +02:00
39d02ab265 Add transcript source kind 2024-10-04 16:38:29 +02:00
ebb32ee613 Transcriptions filtering and search 2024-10-04 13:28:17 +02:00
a85a1ea3d1 Send message if the original is missing 2024-09-06 18:10:38 +02:00
6d976044d0 Update zulip message 2024-09-06 16:09:44 +02:00
a358bfadf1 fix: user.get does not exist 2024-09-03 13:08:36 -06:00
03561453c5 feat: Monadical SSO as replacement of Fief (#393)
* sso: first pass for integrating SSO

still have issue on refreshing
maybe customize the login page, or completely avoid it
make 100% to understand how session server/client are working
need to test with different configuration option (features flags and
requireLogin)

* sso: correctly handle refresh token, with pro-active refresh

Going on interceptors make extra calls to reflector when 401.
We need then to circle back with NextJS backend to update the jwt,
session, then retry the failed request.

I prefered to go pro-active, and ensure the session AND jwt are always
up to date.

A minute before the expiration, we'll try to refresh it. useEffect() of
NextJS cannot be asynchronous, so we cannot wait for the token to be
refreshed.

Every 20s, a minute before the expiration (so 3x in total max) we'll try
to renew. When the accessToken is renewed, the session is updated, and
dispatching up to the client, which updates the useApi().

Therefore, no component will left without a incorrect token.

* fixes: issue with missing key on react-select-search because the default value is undefined

* sso: fixes login/logout button, and avoid seeing the login with authentik page when clicking

* sso: ensure /transcripts/new is not behind protected page, and feature flags page are honored

* sso: fixes user sub->id

* fixes: remove old layout not used

* fixes: set default NEXT_PUBLIC_SITE_URL as localhost

* fixes: removing fief again due to merge with main

* sso: ensure session is always ready before doing any action

* sso: add migration from fief to jwt in server, only from transcripts list

* fixes: user tests

* fixes: compilation issues
2024-09-03 19:27:15 +02:00
f4700f5b8c Remove unused imports 2024-08-22 22:02:07 +02:00
9df5385ff3 Fix transcript link to meeting 2024-08-22 21:46:55 +02:00
35d459a527 Increase meeting length 2024-08-20 12:43:10 +02:00
55697e670d Permanent room urls 2024-08-19 17:56:32 +02:00
2381428ae2 Link recorded meeting to a transcript 2024-08-09 17:30:45 +02:00
Sara
47762e9b05 implement done button 2023-12-13 16:45:49 +01:00
e5d0984363 server: add reviewed in the object output 2023-12-13 15:54:51 +01:00
07b29d42a7 server: add topic duration, and endpoint for getting words group per speaker on a topic 2023-12-11 19:46:05 +01:00
6f3d7df507 server: add API to reassign speakers, and get topics with words 2023-12-06 16:41:18 +01:00
7ac6d25217 server: add participant API
Also break out views into different files for easier reading
2023-11-30 19:13:37 +01:00
Sara
2212d440d4 Merge branch 'main' of github.com:Monadical-SAS/reflector into feat-sharing 2023-11-22 19:28:45 +01:00
0e5c0f66d9 server: move waveform out of the live pipeline 2023-11-22 14:41:40 +01:00
794d08c3a8 server: redirect to storage url 2023-11-22 14:41:40 +01:00
Sara
fe7f1a0e78 Merge branch 'main' of github.com:Monadical-SAS/reflector into feat-sharing 2023-11-21 12:11:58 +01:00
Sara
c40e0970b7 review fixes 2023-11-20 20:13:18 +01:00
Sara
1fc261a669 try to move waveform to pipeline 2023-11-15 20:30:00 +01:00
e18a7c8d4e server: correctly save duration, when filewriter is finished 2023-11-11 01:00:09 +01:00
14946921f3 server: add support for HEAD route on audio mp3 2023-11-10 14:41:21 +01:00
226b92c347 www/server: introduce share mode 2023-11-07 12:39:48 +01:00
6f3e3741e7 hotfix/server: fix crash 2023-11-02 20:12:57 +01:00
aee959369f hotfix/server: correctly load old topic 2023-11-02 20:08:47 +01:00
eb76cd9bcd server/www: rename topic text field to transcript
This aleviate the current issue with vercel deployment
2023-11-02 19:59:56 +01:00
3424550ea9 hotfix/server: add id in GetTranscriptTopic for the frontend to work 2023-11-02 19:06:30 +01:00
d8a842f099 server: full diarization processor implementation based on gokul app 2023-11-02 17:39:21 +01:00
07c4d080c2 server: refactor with diarization, logic works 2023-11-02 17:39:21 +01:00
1c42473da0 server: refactor with clearer pipeline instanciation and linked to model 2023-11-02 17:39:21 +01:00
433c0500cc server: refactor to separate websocket management + start pipeline runner 2023-11-02 17:39:21 +01:00
00c06b7971 server: use redis pubsub for interprocess websocket communication 2023-11-02 17:39:21 +01:00
21e408b323 server: include transcripts words in database, but keep back compatible api 2023-11-02 17:39:21 +01:00
b323254376 server: move out profanity filter to transcript, and implement segmentation 2023-11-02 17:39:21 +01:00
fe6d096f6f server: do not filter empty and recording for now 2023-11-02 12:24:52 +01:00