[#1067] Database Adapter multiple threads#1070
Open
marcocapozzoli wants to merge 21 commits intomasterfrom
Open
[#1067] Database Adapter multiple threads#1070marcocapozzoli wants to merge 21 commits intomasterfrom
marcocapozzoli wants to merge 21 commits intomasterfrom
Conversation
…y and memory management
…e logging level in PostgresWrapper
…sumer count in ThreadPool from 4 to 8 for improved concurrency.
…agement - Added BoundedSharedQueue class with enqueue, dequeue, empty, and size methods. - Updated DatabaseMappingJob, AtomPersistenceJob, and AtomPersistenceJob2 to use BoundedSharedQueue. - Refactored Pipeline and PostgresWrapper to replace SharedQueue with BoundedSharedQueue. - Introduced BatchConsumer for managing batch processing with ThreadPool. - Adjusted logging and batch size configurations for better performance.
… to Atom/Metta transformation - Added DatabaseLoader.h and DatabaseLoader.cc to handle database mapping jobs. - Introduced DatabaseMapper.h and DatabaseMapper.cc for mapping SQL rows to Metta S-Expressions and Atom objects. - Created DatabaseTypes.h to define structures for SQL rows and column values. - Updated PostgresWrapper and DatabaseWrapper to include new database mapping functionalities. - Enhanced main application with a new entry point for database adapter operations. - Updated build configurations to include new source files and dependencies. - Modified test files to integrate new database loader and mapper functionalities.
…ce SQL row processing
andre-senna
reviewed
Apr 14, 2026
| private: | ||
| queue<void*> queue_; | ||
| mutex mtx_; | ||
| condition_variable not_empty_; |
Contributor
There was a problem hiding this comment.
I didn't see where this is used
Collaborator
Author
There was a problem hiding this comment.
You're right, I changed the code and forgot to remove that member
| } | ||
| this->atoms.push_back(atom); | ||
| } | ||
| if (!this->atoms.empty() && this->atoms.size() >= BATCH_SIZE) { |
Contributor
There was a problem hiding this comment.
Suggested change
| if (!this->atoms.empty() && this->atoms.size() >= BATCH_SIZE) { | |
| if (this->atoms.size() >= BATCH_SIZE) { |
| if (this->input_queue->empty()) break; | ||
| auto atom = (Atom*) this->input_queue->dequeue(); | ||
| this->count++; | ||
| if (atom == nullptr) { |
Contributor
There was a problem hiding this comment.
Use nullptr when you make comparison with unique_ptr<type> or shared_ptr<type>.
When using raw pointers (Type*) you should use NULL instead.
| if (this->input_queue->empty()) break; | ||
| auto atom = (Atom*) this->input_queue->dequeue(); | ||
| this->count++; | ||
| if (atom == nullptr) { |
Contributor
There was a problem hiding this comment.
Suggested change
| if (atom == nullptr) { | |
| if (atom == NULL) { |
| try { | ||
| this->atomdb->add_atoms(atoms, false, true); | ||
|
|
||
| auto elapsed = |
Contributor
There was a problem hiding this comment.
Use Utils::StopWatch instead of calling time functions directly.
…unner and BatchConsumer
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #1067