Draft
Conversation
…2246) * Return EventGraph/PersistentGraph correctly in decode_parquet * Remove par_iter from load_node_props_from_df
* trying to cheat on iterators and debug symbols * fix macro for iterators and debug symbols * fix macro for iterators and debug symbols 2 * fix macro for iterators and debug symbols 2 * fix macro for iterators and debug symbols 4 * fix macro for iterators and debug symbols 5 * make box_on_debug a procmacro * the box_on_debug macro works but it generates Boxes without Send + Sync * some progress with the boxed debug symbols # Conflicts: # db4-storage/src/resolver/mod.rs * working debug symbols bypass * symbols are back * removed some useless tests * include all values debug could take
* Fix/top k (#2228) * add linear top_k implementation * update top_k to use new linear top_k implementation * cleanup * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Fix/fastrp (#2229) * update fastrp to properly average embeddings instead of just summing * fix import issue * add helper function to test pairwise distances * cleanup and update python test * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * fix docker ci (#2227) * sort ci action and create new Dockerfile for python * add missing dockerfile * remove env from docker ci action * fix python dockerfile * chore: apply tidy-public auto-fixes * add action input to be able to build the python image * add permissions * read base input * fix typo * change default branch to master * chore: apply tidy-public auto-fixes * fix using wrong username on merge step * change manual docker release to also build python * fix digest name conflicts * change choice to boolean for action input * fix digest name using old variable * sort versioning and add nightly action * add permissions to nightly action * add permissions to manual docker release action * read dry-run from the docker action * add dry_run to docker action --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * graphql bench on CI and vector bench (#2198) * rewrite the vector bench script * try disabling available memory setting * try arroy append api * add graphql benchmarks * chore: apply tidy-public auto-fixes * add parallel workflow for graphql bench * chore: apply tidy-public auto-fixes * setup python * setup k6 * chore: apply tidy-public auto-fixes * get output file out of the results folder * add pnpm-workspace.yaml * fix ci error * chore: apply tidy-public auto-fixes * fix ci for good this time? * test gh pages branch * chore: apply tidy-public auto-fixes * test different dir for graphql * reduce target * add -100 for testing * chore: apply tidy-public auto-fixes * add TODO * chore: apply tidy-public auto-fixes * cleanup PR * re-enable base benches * fix flaky bench * remove -100 * remove cell from results.ipynb * sort parent workflow * add missing bench.ts * chore: apply tidy-public auto-fixes * some final bits * print json output on ci to make sure numbers are valid * fix ci * set iteration target to 6400 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * James/graphql docstrings (#2210) * init * docstrings * docstrings for edges * docstrings for edges * regen schema and docs * run formatting * chore: apply tidy-public auto-fixes * more docstrings * backticks are not used by docs parser so remove * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * more docstrings * update schema and format * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * more docstrings * cleanup * more docstrings * chore: apply tidy-public auto-fixes * testcase for inputs * cleanup * chore: apply tidy-public auto-fixes * more docstrings * cleanup * chore: apply tidy-public auto-fixes * more docstrings * cleanup * chore: apply tidy-public auto-fixes * fix page docstrings * cleanup * remove latin * chore: apply tidy-public auto-fixes * initial fixes * fmt * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Ben Steer <ben.steer@pometry.com> * Release v0.16.1 (#2236) chore: Release Co-authored-by: Pometry-Team <ben.steer@pometry.com> * Fix explode layers for filtered persistent graph (#2241) * explode layers for valid graph is broken * explode_layers was ignoring layer filters for persistent semantics * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * James/graphql docstrings fixes (#2239) * init * docstrings * docstrings for edges * docstrings for edges * regen schema and docs * run formatting * chore: apply tidy-public auto-fixes * more docstrings * backticks are not used by docs parser so remove * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * more docstrings * update schema and format * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * chore: apply tidy-public auto-fixes * more docstrings * more docstrings * cleanup * more docstrings * chore: apply tidy-public auto-fixes * testcase for inputs * cleanup * chore: apply tidy-public auto-fixes * more docstrings * cleanup * chore: apply tidy-public auto-fixes * more docstrings * cleanup * chore: apply tidy-public auto-fixes * fix page docstrings * cleanup * remove latin * chore: apply tidy-public auto-fixes * initial fixes * fmt * chore: apply tidy-public auto-fixes * fix double spacing * review fixes * specify layers for has_edge * tidy * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Ben Steer <ben.steer@pometry.com> Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk> * James/graphql-userguide-16-x (#2233) * update ui image * mutation and views * persistent and event distinction * clean up running steps and add cli * subtitle * proper hierarchy * props and metadata examples * Add troubleshooting * add missing cli parameter * chore: apply tidy-public auto-fixes * add default save location to troubleshooting * markdown formatting * chore: apply tidy-public auto-fixes * chore: apply tidy-public auto-fixes * Clarify docker basics * Clarify docker basics --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk> * fix nightly release action (#2244) * fix nightly release action * fix versions missing * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * add docker retag action (#2245) * add docker retag action * add permissions * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * update Slack invite link (#2252) * test strings in df loaders * Increase sleep time on graphql bench (#2278) Update Makefile * remove polars from parquet and df loading * Bump tracing-subscriber from 0.3.19 to 0.3.20 in the cargo group across 1 directory (#2251) * Bump tracing-subscriber in the cargo group across 1 directory Bumps the cargo group with 1 update in the / directory: [tracing-subscriber](https://github.com/tokio-rs/tracing). Updates `tracing-subscriber` from 0.3.19 to 0.3.20 - [Release notes](https://github.com/tokio-rs/tracing/releases) - [Commits](tokio-rs/tracing@tracing-subscriber-0.3.19...tracing-subscriber-0.3.20) --- updated-dependencies: - dependency-name: tracing-subscriber dependency-version: 0.3.20 dependency-type: direct:production dependency-group: cargo ... Signed-off-by: dependabot[bot] <support@github.com> * chore: apply tidy-public auto-fixes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk> * community detection (#2276) Covers changes for ticket #2268 and adds an introduction to community detection with a tutorial using our existing algorithms and UI. This should apear first in the search results. * initial introduction * add karate example * add karate example * add assets and tests * fix tests * fix tests * fix tests * chore: apply tidy-public auto-fixes * review: simplify node type assignment * chore: apply tidy-public auto-fixes * review: swap to csv file --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Use raphtory from python dir (#2275) * Use raphtory from python dir * comment to match * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * add batch_size argument for the low-level functions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: wyatt-joyner-pometry <wyatt.joyner@pometry.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Pedro Rico Pinazo <ricopinazo@gmail.com> Co-authored-by: James Baross <james.baross@pometry.com> Co-authored-by: Ben Steer <ben.steer@pometry.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk> Co-authored-by: edsherrington <ed.sherrington@pometry.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* small changes to test on actual data * surface earliest and latest * bulk_insert_edges in segment edge * bulk edge loading passes one simple test * write files in relative file format so graphs can be loaded on different boxes * attempt at loading properties faster * default batch size and Externalerror * fixed the edge duplication error but uncovered a node history error * fix node history error, add edge loading test with disk * add check_graph.rs * remove some useless files * minor changes before merge * remove mimalloc * move disk tests away, expose test-utils --------- Co-authored-by: David Moran <david.mrnc@hotmail.com>
…r faster proptest shrinking
no reason to split chunks and view
* Add logging and replay for add_node / create_node * Remove trailing /// * Return NodeWriterT from internal_add_node * Write lsn to node segment * Implement add_node/create_node replay * Bring back node id checking in create_node * Replay node_type * Replay node_type_id * Run fmt * Small cleanup * Use DEFAULT_NODE_TYPE_ID * Remove extra +1 from resize * Cleanup store node info methods * Use resolve_pos methods * Use as_gid_ref().left() * Use add_node_impl and log to wal after acquiring lock * Panic on wal flush failure * Run fmt * Return FatalWriteError on wal flush errors --------- Co-authored-by: Lucas Jeub <lucas.jeub@pometry.com>
* need to check the segment actually has the node * rework of atomic_add_edge to resolve the nodes as needed to avoid leaking dirty nodes * try to fix deadlock when both src and dst are new * fix the deadlock by looping through free segments * fix the deadlock in add_edge by moving initialisation out of the dashmap locks * rework the resolver ops * rework the node additions apis * segment counts for locked graph should only consider locked segments (other segments can be modified) * run stress-test server compiled with debug mode as it is more likely to trigger concurrency issues * fix the stress-test workflow * use high-level api in import for now * use atomic_add_node instead of resolve * layer_id is not needed in atomic_add_edge * need to finish building before starting the stress-test * fix the path * make hits test less sensitive to floating point errors * fix the deadlock in create_node * initialisation read was hanging on to the dashmap lock longer then it should * tweak the segments_iter() such that it waits for uninitialised segments * minor cleanup * need to make sure to use the same locked graph so we don't end up in a weird situation where some segments don't exist yet * make tests independent of node order * fix the slot allocation when picking a segment, add some comments * minor fix in node_store.rs * comment UI tests * fix workflow * Revert "fix workflow" This reverts commit d604f62. * Revert "comment UI tests" This reverts commit 9820891. --------- Co-authored-by: Fabian Murariu <murariu.fabian@gmail.com>
…edge` and graph properties (#2493) * Add logging and replay for add_updates for edges * Remove separate add_edge_properties wal entry We can reuse existing wal code for add_edge * Use set_or_unify_id_and_dtype during replay * Add logging and replay for AddEdgeMetadata * chore: apply tidy-public auto-fixes * Remove src_id and dst_id for add_metadata * Use AddEdgeMetadata logging methods * Add set_lsn method for writers * Use correct mapper for metadata * Create add_metadata_impl and reuse * chore: apply tidy-public auto-fixes * Add wal logging for NodeView.add_updates * Create add_metadata_impl for NodeView and reuse * Add wal logging and replay for NodeView.add_metadata * chore: apply tidy-public auto-fixes * Add wal for set_node_type * Add FIXME for actual set_node_type logging * Add logging and replay for graph_props * Implement set_lsn for GraphPropWriter * Return GraphPropWriterT * Call wal.log in add_properties * Call wal.log in add_properties * Remove some unwraps and collects in replay * Fix some more merge conflicts * Add logging for set_node_type * Run fmt * Run fmt * Cleanup and fix reserve_and_lock_segment * Remove Option from WriteLockedGraphPropPages writer * Check lsn during graph prop replay * Refactor graph props add_metadata * Implement logging and replay for graph metadata * Use is_update correctly * Implement logging and replay for delete_edge * Run fmt * Simplify some comments * Run fmt * fix some review comments * fmt * Add missing ? on unify_types * Update comment * Add new nodes into resolver in replay_delete_edge * Fix src, dst and edge writes in replay_delete_edge * Use STATIC_GRAPH_LAYER_ID * Add wal logging for EdgeView.delete_edge * Run fmt * Cleanup some config --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Fabian Murariu <murariu.fabian@gmail.com>
* update ui tests * Update Playwright installation command in ui test workflow * Update UI test workflow with timeout and install changes Increase timeout for UI tests. * Update index.html * Set ui-tests back to fix/v4 branch * revert timeout mins * chore: apply tidy-public auto-fixes --------- Co-authored-by: Rachel Chan <25484244+rachchan@users.noreply.github.com> Co-authored-by: Louis Chan <louisch@fastmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Cleanup wal on clean shutdown * Cleanup wal file on graceful shutdown * Run fmt * Rename to latest_immut_lsn * Set immut_lsn to return 0 for no-disk * Add checkpoint after rotate * Use latest_lsn_on_disk = wal.next_lsn() - 1 * Revert setting LSN = 1 for NoWal --------- Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk>
* Made the add_node, add_edge, create_node, delete_edge functions on GraphQLRemoteGraph return the appropriate node/edge instead of (). Added a remote_graph function on RaphtoryGraphQLClient. * Fixed rust graphql client bugs * Trying to find out why concurrent fault injector is failing * Changed RaphtoryGraphQLClient to hold a url::Url instead of a String and added a health_check on it. Moved OnceLock tokio runtime from python submodule to raphtory crate so we can use it in the RaphtoryGraphQLClient and fault-injector. Updating fault-injector to use this properly. Addressing PR comments. * Changed RaphtoryGraphQLClient::new token to be Option instead of String * Moved the static tokio runtime back to the python submodule, and added a different one for use in the fault-injector. * Addressed PR comments * Fixed failing python tests because of DTime and NDTime being supported in graphql now instead of writing them as strings. The datetimes have to be truncated to millisecond precision to match the graphql server.
* v1 * gql for v4 * more clams * clamin * allocator issues * latest clam * optional match * update clam-core: standalone RETURN + exponentiation operator * update clam-core: modulo + string operators * update clam-core: add 15+ functions to binder passthrough * plenty of cypher fixes * make submodules use ssh * Update clam-core: XOR and IN operator support * Update clam-core: duration function 2-arg overload * cypher fixees * update submodule * update submodule * cypher * update clam-core post-merge * implement efficient skip for StateIndex iterator * update submodule * make arc_into_iter efficient as well * add support for returning index to node list * concrete type for index iterator * node scan no longer eagerly filters and collects the node ids * update submodule * Start fixing the graph apis in the query execution (#2497) * update submodule * update submodule * update clam-core post-merge * implement efficient skip for StateIndex iterator * update submodule * make arc_into_iter efficient as well * add support for returning index to node list * concrete type for index iterator * node scan no longer eagerly filters and collects the node ids * update submodule * new clam * clamza * update submodule * new clam * update clam-core * new clam * new clams * update clam-core * merge clam-core * custom implementation of take that is updateable * avoid allocation in edge scan * optd update missing after merge * minor refactor of StateIndex to support IntoIterator * some more Layer inputs are now allowed * cleanup and update clam-core * update clam-core * update clam-core * optimise is_empty for edges * some into's are no longer needed * more into's removed * fix silly bug in is_empty * update clam-core * new clam * Updating to new clam * new clams * Db v4 gql avoid collect (#2501) update Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk> * new clam * Make expand preserve batch sizes correctly (#2503) * new clam-core * new clam-core * load segments in parallel and avoid resolver.len call * cleanup warnings # Conflicts: # raphtory/src/db/api/storage/storage.rs * start work on implementing fast path for layer filtering everywhere * fix node_count * pass down layer_ids for filtering for the node storage operations * need to handle node property additions properly * fix degree computation in the storage layer for multiple layers * fix and simplify the counting of edges per layer * layered graph needs to invalidate previously filtered node lists that were trusted * enable ci tests * cleanup * fmt * check if a row has any valid values * move fast_merge to raphtory-itertools * need to check for null datatype * some efficiency improvements for materialize * remove println * update the node page size in tests * Fix/parallel exec (#2509) * update clam-core * add cargo lock * Implement wal logging and replay for `EdgeView`, `NodeView`, `delete_edge` and graph properties (#2493) * Add logging and replay for add_updates for edges * Remove separate add_edge_properties wal entry We can reuse existing wal code for add_edge * Use set_or_unify_id_and_dtype during replay * Add logging and replay for AddEdgeMetadata * chore: apply tidy-public auto-fixes * Remove src_id and dst_id for add_metadata * Use AddEdgeMetadata logging methods * Add set_lsn method for writers * Use correct mapper for metadata * Create add_metadata_impl and reuse * chore: apply tidy-public auto-fixes * Add wal logging for NodeView.add_updates * Create add_metadata_impl for NodeView and reuse * Add wal logging and replay for NodeView.add_metadata * chore: apply tidy-public auto-fixes * Add wal for set_node_type * Add FIXME for actual set_node_type logging * Add logging and replay for graph_props * Implement set_lsn for GraphPropWriter * Return GraphPropWriterT * Call wal.log in add_properties * Call wal.log in add_properties * Remove some unwraps and collects in replay * Fix some more merge conflicts * Add logging for set_node_type * Run fmt * Run fmt * Cleanup and fix reserve_and_lock_segment * Remove Option from WriteLockedGraphPropPages writer * Check lsn during graph prop replay * Refactor graph props add_metadata * Implement logging and replay for graph metadata * Use is_update correctly * Implement logging and replay for delete_edge * Run fmt * Simplify some comments * Run fmt * fix some review comments * fmt * Add missing ? on unify_types * Update comment * Add new nodes into resolver in replay_delete_edge * Fix src, dst and edge writes in replay_delete_edge * Use STATIC_GRAPH_LAYER_ID * Add wal logging for EdgeView.delete_edge * Run fmt * Cleanup some config --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Fabian Murariu <murariu.fabian@gmail.com> # Conflicts: # db4-storage/src/pages/edge_page/writer.rs # raphtory/src/db/api/storage/storage.rs * cleanup warnings # Conflicts: # raphtory/src/db/api/storage/storage.rs * Cargo.lock * clam-core * set version to 0.17.0 for clam-core * post rebase * add recursive checkout with token * update all actions/checkout@v5 * disable jemalloc seems to cause issues * disable jemalloc seems to cause issues * Cargo.lock * fix io imports * create progress feature * fix index creation for layered graph * fix compilation issues * update submodules * fmt * remove old branches from workflow * correct ui tests * should avoid trying to use the pull token for push later * temporarily disable needs for debugging * new attempt with auth * chore: apply tidy-public auto-fixes * bring back needs checks to avoid green ticking * fmt --------- Co-authored-by: miratepuffin <b.a.steer@qmul.ac.uk> Co-authored-by: Fabian Murariu <murariu.fabian@gmail.com> Co-authored-by: Shivam <4599890+shivamka1@users.noreply.github.com> Co-authored-by: Fadhil Abubaker <fadhil.abubaker@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Remove default implementation of load_with_config * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* v1 * gql for v4 * more clams * clamin * allocator issues * latest clam * optional match * update clam-core: standalone RETURN + exponentiation operator * update clam-core: modulo + string operators * update clam-core: add 15+ functions to binder passthrough * plenty of cypher fixes * make submodules use ssh * Update clam-core: XOR and IN operator support * Update clam-core: duration function 2-arg overload * cypher fixees * update submodule * update submodule * cypher * update clam-core post-merge * implement efficient skip for StateIndex iterator * update submodule * make arc_into_iter efficient as well * add support for returning index to node list * concrete type for index iterator * node scan no longer eagerly filters and collects the node ids * update submodule * Start fixing the graph apis in the query execution (#2497) * update submodule * update submodule * update clam-core post-merge * implement efficient skip for StateIndex iterator * update submodule * make arc_into_iter efficient as well * add support for returning index to node list * concrete type for index iterator * node scan no longer eagerly filters and collects the node ids * update submodule * new clam * clamza * update submodule * new clam * update clam-core * new clam * new clams * update clam-core * merge clam-core * custom implementation of take that is updateable * avoid allocation in edge scan * optd update missing after merge * minor refactor of StateIndex to support IntoIterator * some more Layer inputs are now allowed * cleanup and update clam-core * update clam-core * update clam-core * optimise is_empty for edges * some into's are no longer needed * more into's removed * fix silly bug in is_empty * update clam-core * new clam * Updating to new clam * new clams * Db v4 gql avoid collect (#2501) update Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk> * new clam * Make expand preserve batch sizes correctly (#2503) * new clam-core * new clam-core * load segments in parallel and avoid resolver.len call * cleanup warnings # Conflicts: # raphtory/src/db/api/storage/storage.rs * start work on implementing fast path for layer filtering everywhere * fix node_count * pass down layer_ids for filtering for the node storage operations * need to handle node property additions properly * fix degree computation in the storage layer for multiple layers * fix and simplify the counting of edges per layer * layered graph needs to invalidate previously filtered node lists that were trusted * enable ci tests * cleanup * fmt * check if a row has any valid values * move fast_merge to raphtory-itertools * need to check for null datatype * some efficiency improvements for materialize * remove println * update the node page size in tests * Fix/parallel exec (#2509) * update clam-core * add cargo lock * Implement wal logging and replay for `EdgeView`, `NodeView`, `delete_edge` and graph properties (#2493) * Add logging and replay for add_updates for edges * Remove separate add_edge_properties wal entry We can reuse existing wal code for add_edge * Use set_or_unify_id_and_dtype during replay * Add logging and replay for AddEdgeMetadata * chore: apply tidy-public auto-fixes * Remove src_id and dst_id for add_metadata * Use AddEdgeMetadata logging methods * Add set_lsn method for writers * Use correct mapper for metadata * Create add_metadata_impl and reuse * chore: apply tidy-public auto-fixes * Add wal logging for NodeView.add_updates * Create add_metadata_impl for NodeView and reuse * Add wal logging and replay for NodeView.add_metadata * chore: apply tidy-public auto-fixes * Add wal for set_node_type * Add FIXME for actual set_node_type logging * Add logging and replay for graph_props * Implement set_lsn for GraphPropWriter * Return GraphPropWriterT * Call wal.log in add_properties * Call wal.log in add_properties * Remove some unwraps and collects in replay * Fix some more merge conflicts * Add logging for set_node_type * Run fmt * Run fmt * Cleanup and fix reserve_and_lock_segment * Remove Option from WriteLockedGraphPropPages writer * Check lsn during graph prop replay * Refactor graph props add_metadata * Implement logging and replay for graph metadata * Use is_update correctly * Implement logging and replay for delete_edge * Run fmt * Simplify some comments * Run fmt * fix some review comments * fmt * Add missing ? on unify_types * Update comment * Add new nodes into resolver in replay_delete_edge * Fix src, dst and edge writes in replay_delete_edge * Use STATIC_GRAPH_LAYER_ID * Add wal logging for EdgeView.delete_edge * Run fmt * Cleanup some config --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Fabian Murariu <murariu.fabian@gmail.com> # Conflicts: # db4-storage/src/pages/edge_page/writer.rs # raphtory/src/db/api/storage/storage.rs * cleanup warnings # Conflicts: # raphtory/src/db/api/storage/storage.rs * Cargo.lock * clam-core * set version to 0.17.0 for clam-core * post rebase * add recursive checkout with token * update all actions/checkout@v5 * disable jemalloc seems to cause issues * disable jemalloc seems to cause issues * Cargo.lock * fix io imports * create progress feature * fix index creation for layered graph * fix compilation issues * update submodules * fmt * remove old branches from workflow * correct ui tests * remove unused methods to make it easier to implement resolvers * add flushing for resolvers * show progress for edges * expose resolver in core ops * minor changes in resolver API * chore: apply tidy-public auto-fixes * global estimation for nodes * add global flush in bulk loader * small changes to support faster bulk loading * global memory tracking for edges * need to flush node segments as well during edge loading * make N in Inner storage depend on available parallelism * move progress bar to progress feature * cleanup after merge * bring back the dropping of the write-locked graph * track memory on drop * delete weird stub file * should flush node segments as well * add pause * pause in bulk loaders * fmt * debug * debug for local flush as well * don't make as many writers * fix borrow-checker issues * fix the writer position * fmt * pause early * try to simplify the flushing logic and move all the writers outside the loop * fmt * set the number of free segments based on the rayon number of threads which is more relevant and can be changed by environment variable for testing * more testing of the skip for the index iterator * fix bug in the index iterator where calling nth past the end of the iterator would not exhaust the iterator * make sure we always generate a non-empty subgraph * improve the PathFromGraph api to return the source node * make the type filter test node-order invariant * ignore new nodes when loading metadata * minor fixes for load node props * improve PathFromGraph apis and make tests independent of node order * clean up a lot of warnings * make the rest of the rust tests order-independent * fmt * add floating point tolerance to hits test * most python tests are order-independent, some problem with nested sorting still * fix the sorting * simplify the sorting test so it doesn't depend on the node order * some more python tests that require sorting --------- Co-authored-by: miratepuffin <b.a.steer@qmul.ac.uk> Co-authored-by: Fabian Murariu <murariu.fabian@gmail.com> Co-authored-by: Shivam <4599890+shivamka1@users.noreply.github.com> Co-authored-by: Fadhil Abubaker <fadhil.abubaker@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Fabian Murariu <2404621+fabianmurariu@users.noreply.github.com>
* add required components to rust toolchain action * don't require ui tests
* trying to get rid of slots and the dashmap iterator * various changes to bulk loading to speed-up loading nodes and edges * remove some inlines and return the correct error for node type * internal deletion ops are not used anymore * remove test-only apis and associated tests * fixing node type missalignment * fmt * chore: apply tidy-public auto-fixes * only flush edge segments when they are disty on bulk loads --------- Co-authored-by: Lucas Jeub <lucas.jeub@pometry.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: David Moran <david.mrnc@hotmail.com>
* optimise has_edge checks * update optd
* remove static lcc * fmt
* add test * fixed order for resolving node id pairs in atomic add_edge to avoid deadlocks * fmt
* start work on arrow-based string columns * string column compiles * AsPropRef * make writers take PropRef * need to actually add the updated values to the column * add a proptest for StringCol * move to PropRef when updating graph, avoid str clone * in some tests we get back pyarrow * fix python tests * chore: apply tidy-public auto-fixes --------- Co-authored-by: Fabian Murariu <murariu.fabian@gmail.com> Co-authored-by: Fabian Murariu <2404621+fabianmurariu@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* remove static lcc * fmt * new node state macro * fmt
* fix benchmark cve * fix cve raphtory
* Build(deps): bump actions/setup-node from 4 to 6 (#2534) Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 6. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v4...v6) --- updated-dependencies: - dependency-name: actions/setup-node dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build(deps): bump peter-evans/create-pull-request from 5 to 8 (#2535) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 5 to 8. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v5...v8) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build(deps): bump webfactory/ssh-agent from 0.9.0 to 0.10.0 (#2536) Bumps [webfactory/ssh-agent](https://github.com/webfactory/ssh-agent) from 0.9.0 to 0.10.0. - [Release notes](https://github.com/webfactory/ssh-agent/releases) - [Changelog](https://github.com/webfactory/ssh-agent/blob/master/CHANGELOG.md) - [Commits](webfactory/ssh-agent@v0.9.0...v0.10.0) --- updated-dependencies: - dependency-name: webfactory/ssh-agent dependency-version: 0.10.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build(deps): bump pnpm/action-setup from 4 to 5 (#2537) Bumps [pnpm/action-setup](https://github.com/pnpm/action-setup) from 4 to 5. - [Release notes](https://github.com/pnpm/action-setup/releases) - [Commits](pnpm/action-setup@v4...v5) --- updated-dependencies: - dependency-name: pnpm/action-setup dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build(deps): bump geekyeggo/delete-artifact from 2 to 6 (#2538) Bumps [geekyeggo/delete-artifact](https://github.com/geekyeggo/delete-artifact) from 2 to 6. - [Release notes](https://github.com/geekyeggo/delete-artifact/releases) - [Changelog](https://github.com/GeekyEggo/delete-artifact/blob/main/CHANGELOG.md) - [Commits](GeekyEggo/delete-artifact@v2...v6) --- updated-dependencies: - dependency-name: geekyeggo/delete-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build(deps): bump the npm_and_yarn group across 1 directory with 8 updates (#2543) Bumps the npm_and_yarn group with 7 updates in the /graphql-bench directory: | Package | From | To | | --- | --- | --- | | [tsup](https://github.com/egoist/tsup) | `8.3.0` | `8.3.5` | | [lodash](https://github.com/lodash/lodash) | `4.17.21` | `4.17.23` | | [minimatch](https://github.com/isaacs/minimatch) | `9.0.5` | `9.0.9` | | [minimatch](https://github.com/isaacs/minimatch) | `3.1.2` | `3.1.5` | | [picomatch](https://github.com/micromatch/picomatch) | `2.3.1` | `2.3.2` | | [picomatch](https://github.com/micromatch/picomatch) | `4.0.3` | `4.0.4` | | [qs](https://github.com/ljharb/qs) | `6.14.1` | `6.15.0` | | [rollup](https://github.com/rollup/rollup) | `4.55.1` | `4.60.0` | | [@tootallnate/once](https://github.com/TooTallNate/once) | `2.0.0` | `removed` | Updates `tsup` from 8.3.0 to 8.3.5 - [Release notes](https://github.com/egoist/tsup/releases) - [Commits](egoist/tsup@v8.3.0...v8.3.5) Updates `esbuild` from 0.23.1 to 0.24.2 - [Release notes](https://github.com/evanw/esbuild/releases) - [Changelog](https://github.com/evanw/esbuild/blob/main/CHANGELOG-2024.md) - [Commits](evanw/esbuild@v0.23.1...v0.24.2) Updates `lodash` from 4.17.21 to 4.17.23 - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.21...4.17.23) Updates `minimatch` from 9.0.5 to 9.0.9 - [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md) - [Commits](isaacs/minimatch@v9.0.5...v9.0.9) Updates `minimatch` from 3.1.2 to 3.1.5 - [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md) - [Commits](isaacs/minimatch@v9.0.5...v9.0.9) Updates `picomatch` from 2.3.1 to 2.3.2 - [Release notes](https://github.com/micromatch/picomatch/releases) - [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md) - [Commits](micromatch/picomatch@2.3.1...2.3.2) Updates `picomatch` from 4.0.3 to 4.0.4 - [Release notes](https://github.com/micromatch/picomatch/releases) - [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md) - [Commits](micromatch/picomatch@2.3.1...2.3.2) Updates `qs` from 6.14.1 to 6.15.0 - [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md) - [Commits](ljharb/qs@v6.14.1...v6.15.0) Updates `rollup` from 4.55.1 to 4.60.0 - [Release notes](https://github.com/rollup/rollup/releases) - [Changelog](https://github.com/rollup/rollup/blob/master/CHANGELOG.md) - [Commits](rollup/rollup@v4.55.1...v4.60.0) Removes `@tootallnate/once` --- updated-dependencies: - dependency-name: tsup dependency-version: 8.3.5 dependency-type: direct:development dependency-group: npm_and_yarn - dependency-name: esbuild dependency-version: 0.24.2 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: lodash dependency-version: 4.17.23 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: minimatch dependency-version: 9.0.9 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: minimatch dependency-version: 3.1.5 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: picomatch dependency-version: 2.3.2 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: picomatch dependency-version: 4.0.4 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: qs dependency-version: 6.15.0 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: rollup dependency-version: 4.60.0 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: "@tootallnate/once" dependency-version: dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * jsonwebtoken = "10.3.0" * jwt cve * chore: apply tidy-public auto-fixes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* unlock the existing full segment before finding free segment for the other node when needed * clean up warnings * add back missing import * simplify reserve_and_lock_segment for new assumptions * move comment
* add DSU connected components algo * bugfix and optimize dsu cc * chore: apply tidy-public auto-fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Progress towards the new version of the underlying storage
Why are the changes needed?
Does this PR introduce any user-facing change? If yes is this documented?
How was this patch tested?
Are there any further changes required?