diff --git a/docs/docs/getting_started/installation.md b/docs/docs/getting_started/installation.md index f5d842a0..827c6e2f 100644 --- a/docs/docs/getting_started/installation.md +++ b/docs/docs/getting_started/installation.md @@ -5,9 +5,9 @@ ### Prerequisites - Docker and Docker Compose - Go 1.19+ (for CORE service) -- Ballerina (for APIs) +- Ballerina 2201.13.4 (for APIs) -### Start the System +### Start the System (Using Docker) 1. **Start databases** ```bash @@ -23,6 +23,95 @@ docker-compose up -d ingestion read ``` +### Start the System (Locally, without Docker) + +1. **Start databases** — choose one: + + **Option A: Local databases (Docker)** + ```bash + docker-compose up -d mongodb neo4j postgres + ``` + + **Option B: Cloud databases** (e.g. [Neo4j Aura](https://neo4j.com/cloud/aura/), [MongoDB Atlas](https://www.mongodb.com/atlas), a managed PostgreSQL instance) + + Skip the Docker step above and instead point core-api at your cloud instances by setting the connection variables in `opengin/core-api/.env` (or `.env.ps1` on Windows) before starting CORE in step 3: + ```bash + export NEO4J_URI=neo4j+s://.databases.neo4j.io + export NEO4J_USER=neo4j + export NEO4J_PASSWORD= + export MONGO_URI= + export POSTGRES_HOST= + export POSTGRES_PORT=5432 + export POSTGRES_USER= + export POSTGRES_PASSWORD= + export POSTGRES_DB= + ``` + (Windows PowerShell: same variables using `$env:NEO4J_URI="..."` syntax, in `.env.ps1`.) + + Note Neo4j Aura uses the `neo4j+s://` scheme (encrypted), not `bolt://` — using `bolt://` against an Aura instance will fail to connect. + +2. **Start CORE service** + + For LINUX & macOS + ```bash + cd opengin/core-api + cp env.template .env + source .env + go build -o core-service cmd/server/service.go cmd/server/utils.go + ./core-service + ``` + + For Windows (PowerShell) + ```powershell + cd opengin/core-api + Copy-Item env.template.ps1 .env.ps1 + . .\.env.ps1 + go build -o core-service.exe cmd/server/service.go cmd/server/utils.go + .\core-service.exe + ``` + + See [core-api/README.md](../../../opengin/core-api/README.md) for details. + +3. **Start ingestion-api** (make sure CORE service is running) + + For LINUX & macOS + ```bash + cd opengin/ingestion-api + cp env.template .env + source .env + bal run + ``` + + For Windows (PowerShell) + ```powershell + cd opengin/ingestion-api + Copy-Item env.template.ps1 .env.ps1 + . .\.env.ps1 + bal run + ``` + + See [ingestion-api/README.md](../../../opengin/ingestion-api/README.md) for details. + +4. **Start read-api** (make sure CORE service is running) + + For LINUX & macOS + ```bash + cd opengin/read-api + cp env.template .env + source .env + bal run + ``` + + For Windows (PowerShell) + ```powershell + cd opengin/read-api + Copy-Item env.template.ps1 .env.ps1 + . .\.env.ps1 + bal run + ``` + + See [read-api/README.md](../../../opengin/read-api/README.md) for details. + ### Test the System **Run E2E tests** diff --git a/opengin/core-api/README.md b/opengin/core-api/README.md index 9c3c357a..23b7e023 100644 --- a/opengin/core-api/README.md +++ b/opengin/core-api/README.md @@ -33,7 +33,18 @@ source .env # make sure to clean the dbs before running the tests go test -v ./... -count=1 -p=1 ./core-service +``` +For Windows (PowerShell) +```powershell +cd core-api +Copy-Item env.template.ps1 .env.ps1 +# after updating the required fields to be added to the environment +. .\.env.ps1 +# make sure to clean the dbs before running the tests +go test -v ./... -count=1 -p=1 +.\core-service.exe +``` ## Go Module Setup @@ -115,10 +126,16 @@ docker run -d \ #### Validate +For LINUX & macOS ```bash brew install grpcurl ``` +For Windows, install grpcurl via Chocolatey: +```powershell +choco install grpcurl +``` + ```bash grpcurl -plaintext localhost:50051 list ``` diff --git a/opengin/core-api/env.template.ps1 b/opengin/core-api/env.template.ps1 new file mode 100644 index 00000000..7c646ff7 --- /dev/null +++ b/opengin/core-api/env.template.ps1 @@ -0,0 +1,28 @@ +$env:MONGO_URI="" +$env:MONGO_DB_NAME="" +$env:MONGO_COLLECTION="" + +## Uncomment the following for development + +# $env:NEO4J_URI=$env:NEO4J_DEVELOPMENT_DB_URI +# $env:NEO4J_USER=$env:NEO4J_DEVELOPMENT_USERNAME +# $env:NEO4J_PASSWORD=$env:NEO4J_DEVELOPMENT_PASSWORD + +## If commented, uncomment the following will be used for testing + +# $env:NEO4J_URI=$env:NEO4J_TESTING_DB_URI +# $env:NEO4J_USER=$env:NEO4J_TESTING_USERNAME +# $env:NEO4J_PASSWORD=$env:NEO4J_TESTING_PASSWORD + +## PostgreSQL configuration + +$env:POSTGRES_HOST="localhost" +$env:POSTGRES_PORT="5432" +$env:POSTGRES_USER="postgres" +$env:POSTGRES_PASSWORD="postgres" +$env:POSTGRES_DB="opengin" +$env:POSTGRES_SSL_MODE="disable" +$env:POSTGRES_TEST_DB_URI="postgresql://postgres:postgres@localhost:5432/opengin?sslmode=disable" + +$env:CORE_SERVICE_HOST="localhost" +$env:CORE_SERVICE_PORT="50051" diff --git a/opengin/ingestion-api/README.md b/opengin/ingestion-api/README.md index 3e6cdb5b..b1141a26 100644 --- a/opengin/ingestion-api/README.md +++ b/opengin/ingestion-api/README.md @@ -1,56 +1,78 @@ # Ingestion API -## Generate Open API +## Prerequisites -This will generate the endpoints for the Ingestion API server using the OpenAPI specification. +⚠️ **Warning** +You should have the Ballerina version of **2201.11.0** in your device. You can download it by following link +```bash +https://ballerina.io/downloads/archived/#swan-lake-archived-versions +``` + +## Generate OpenAPI Service + +This will generate the endpoints for the Ingestion API server using the OpenAPI specification. The OpenAPI specification is the base for public API for Ingestion API. > 💡 Note: Always make sure the contract has the expected endpoints and request params -> before working on the code. The generated endpoints should not be editable at all. +> before working on the code. The generated endpoints should not be editable at all. > Maybe the only changes that can be done is adding error handlers, but request and response -> must be defined in the contract. - +> must be defined in the contract. +> Note that this will create the update_api_service_copy.bal file ```bash bal openapi -i ../contracts/rest/ingestion_api.yaml --mode service ``` -## Generate GRPC Stubs +## Generate gRPC Stubs -The client stub generated here will be sending and receiving values via Grpc. -This will send requests to the corresponding CORE server endpoint. +The client stub generated here will be sending and receiving values via Grpc. +This will send requests to the corresponding CORE server endpoint. +This command will generate the types_v1_pb.bal file ```bash bal grpc --mode client --input ../core-api/protos/types_v1.proto --output . ``` > 💡 **Note** -> At the generation make sure to remove any sample code generated to show how to use the API. Because that might add an unnecessary main file. +> At the generation make sure to remove any sample code generated to show how to use the API. Because that might add an unnecessary main file. -## Set Environmental Variables +## Environment Variables -Following are the default values you should use. +For LINUX & macOS +```bash +cd opengin/ingestion-api +cp env.template .env +# update the required fields to set the environment variables +source .env +``` +For Windows (PowerShell) +```powershell +cd opengin/ingestion-api +Copy-Item env.template.ps1 .env.ps1 +# after updating the required fields to be added to the environment +. .\.env.ps1 +``` + +Example .env configuration ```bash +# For LINUX & macOS export CORE_SERVICE_HOST=localhost export CORE_SERVICE_PORT=50051 export INGESTION_SERVICE_HOST=localhost export INGESTION_SERVICE_PORT=8080 -``` - -## Development -```bash -cd ingestion-api -cp env.template .env -# update the required fields to set the environment variables -source .env -bal test +# For Windows (paste on Powershell CLI) +$env:CORE_SERVICE_HOST="localhost" +$env:CORE_SERVICE_PORT="50051" +$env:INGESTION_SERVICE_HOST="localhost" +$env:INGESTION_SERVICE_PORT="8080" ``` -## Run Test +## Run Tests -Make sure the CORE server is running. (`cd design/core-api; ./core-server`) +Make sure the CORE server is running. +(LINUX & macOS: `cd opengin/core-api && ./core-service`; Windows: `cd opengin/core-api; .\core-service.exe`) ```bash # Run all tests in the current package @@ -72,9 +94,6 @@ bal test --code-coverage ## Run Service ```bash -cd ingestion-api +cd opengin/ingestion-api bal run ``` - -At the moment the port is hardcoded to 8080. This must be configurable via a config file. - diff --git a/opengin/ingestion-api/env.template.ps1 b/opengin/ingestion-api/env.template.ps1 new file mode 100644 index 00000000..17138548 --- /dev/null +++ b/opengin/ingestion-api/env.template.ps1 @@ -0,0 +1,11 @@ +# config format 1 (default for development) +$env:CORE_SERVICE_HOST="0.0.0.0" +$env:CORE_SERVICE_PORT="50051" +$env:INGESTION_SERVICE_HOST="0.0.0.0" +$env:INGESTION_SERVICE_PORT="8080" +$env:CORE_SERVICE_URL="http://localhost:50051" + +# config format 2 (default for deployment) + +$env:CORE_SERVICE_URL="http://0.0.0.0:50051" +$env:INGESTION_SERVICE_URL="http://0.0.0.0:8080" diff --git a/opengin/read-api/README.md b/opengin/read-api/README.md index d0ec6369..5cb75b5c 100644 --- a/opengin/read-api/README.md +++ b/opengin/read-api/README.md @@ -1,6 +1,6 @@ # Read API -## Implement OpenAPI Contract +## Prerequisites ⚠️ **Warning** You should have the Ballerina version of **2201.11.0** in your device. You can download it by following link @@ -8,58 +8,53 @@ You should have the Ballerina version of **2201.11.0** in your device. You can d https://ballerina.io/downloads/archived/#swan-lake-archived-versions ``` -The Ballerina service can be generated by the following command. +## Generate OpenAPI Service + +This will generate the endpoints for the Read API server using the OpenAPI specification. +The OpenAPI specification is the base for public API for Read API. > 💡 Note: Always make sure the contract has the expected endpoints and request params -> before working on the code. The generated endpoints should not be editable at all. +> before working on the code. The generated endpoints should not be editable at all. > Maybe the only changes that can be done is adding error handlers, but request and response -> must be defined in the contract. +> must be defined in the contract. > Note that this will create the read_api_service.bal file and types.bal file ```bash bal openapi -i ../contracts/rest/read_api.yaml --mode service ``` -## Generate GRPC Stubs +## Generate gRPC Stubs -The client stub generated here will be sending and receiving values via Grpc. -This will send requests to the corresponding CORE server endpoint. +The client stub generated here will be sending and receiving values via Grpc. +This will send requests to the corresponding CORE server endpoint. This command will generate the types_v1_pb.bal file ```bash bal grpc --mode client --input ../core-api/protos/types_v1.proto --output . ``` -## Development +> 💡 **Note** +> At the generation make sure to remove any sample code generated to show how to use the API. Because that might add an unnecessary main file. -⚠️ **Warning** -All the commands here are for the **LINUX** & **macOS**, but they work on the **Windows** too. but for the commands which does not works on the **Windows** We have given the working command by mentioning that. Please look for that. +## Environment Variables For LINUX & macOS ```bash cd opengin/read-api cp env.template .env # update the required fields to set the environment variables -# (you can find the example env configurations here) source .env -# make sure core service is running -bal test -bal run ``` -For windows (make sure you open the **Powershell CLI**) -```bash +For Windows (PowerShell) +```powershell cd opengin/read-api -echo .env +Copy-Item env.template.ps1 .env.ps1 # after updating the required fields to be added to the environment -# you have to copy and paste the env configurations on the Powershell CLI -# (you can find the example env configurations here) -# make sure core service is running -bal test -bal run +. .\.env.ps1 ``` -Example .env configurations +Example .env configuration ```bash # For LINUX & macOS export CORE_SERVICE_HOST=localhost @@ -74,4 +69,31 @@ $env:READ_SERVICE_HOST="localhost" $env:READ_SERVICE_PORT="8081" ``` +## Run Tests + +Make sure the CORE server is running. +(LINUX & macOS: `cd opengin/core-api && ./core-service`; Windows: `cd opengin/core-api; .\core-service.exe`) + +```bash +# Run all tests in the current package +bal test +# Run tests with verbose output +bal test --test-report + +# Run a specific test file +bal test tests/read_api_service_test.bal + +# Run a specific test function +bal test --tests testEntityAttributeRetrieval + +# Run tests and generate a coverage report +bal test --code-coverage +``` + +## Run Service + +```bash +cd opengin/read-api +bal run +``` diff --git a/opengin/read-api/env.template.ps1 b/opengin/read-api/env.template.ps1 new file mode 100644 index 00000000..bba283ec --- /dev/null +++ b/opengin/read-api/env.template.ps1 @@ -0,0 +1,7 @@ +$env:CORE_SERVICE_HOST="0.0.0.0" +$env:CORE_SERVICE_PORT="50051" +$env:READ_SERVICE_HOST="0.0.0.0" +$env:READ_SERVICE_PORT="8081" + +$env:CORE_SERVICE_URL="http://0.0.0.0:50051" +$env:READ_SERVICE_URL="http://0.0.0.0:8081"