Skip to content
93 changes: 91 additions & 2 deletions docs/docs/getting_started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
### Prerequisites
- Docker and Docker Compose
- Go 1.19+ (for CORE service)
- Ballerina (for APIs)
- Ballerina 2201.13.4 (for APIs)

### Start the System
### Start the System (Using Docker)
Comment thread
coderabbitai[bot] marked this conversation as resolved.

1. **Start databases**
```bash
Expand All @@ -23,6 +23,95 @@
docker-compose up -d ingestion read
```

### Start the System (Locally, without Docker)
Comment thread
coderabbitai[bot] marked this conversation as resolved.

1. **Start databases** — choose one:

**Option A: Local databases (Docker)**
```bash
docker-compose up -d mongodb neo4j postgres
```

**Option B: Cloud databases** (e.g. [Neo4j Aura](https://neo4j.com/cloud/aura/), [MongoDB Atlas](https://www.mongodb.com/atlas), a managed PostgreSQL instance)

Skip the Docker step above and instead point core-api at your cloud instances by setting the connection variables in `opengin/core-api/.env` (or `.env.ps1` on Windows) before starting CORE in step 3:
Comment thread
coderabbitai[bot] marked this conversation as resolved.
```bash
export NEO4J_URI=neo4j+s://<your-instance-id>.databases.neo4j.io
export NEO4J_USER=neo4j
export NEO4J_PASSWORD=<your-password>
export MONGO_URI=<your-mongodb-atlas-connection-string>
export POSTGRES_HOST=<your-postgres-host>
export POSTGRES_PORT=5432
export POSTGRES_USER=<your-postgres-user>
export POSTGRES_PASSWORD=<your-postgres-password>
export POSTGRES_DB=<your-postgres-db>
```
Comment thread
NethmikaKekuu marked this conversation as resolved.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
(Windows PowerShell: same variables using `$env:NEO4J_URI="..."` syntax, in `.env.ps1`.)

Note Neo4j Aura uses the `neo4j+s://` scheme (encrypted), not `bolt://` — using `bolt://` against an Aura instance will fail to connect.

2. **Start CORE service**

For LINUX & macOS
```bash
cd opengin/core-api
cp env.template .env
source .env
go build -o core-service cmd/server/service.go cmd/server/utils.go
./core-service
```

For Windows (PowerShell)
```powershell
cd opengin/core-api
Copy-Item env.template.ps1 .env.ps1
. .\.env.ps1
go build -o core-service.exe cmd/server/service.go cmd/server/utils.go
.\core-service.exe
Comment thread
coderabbitai[bot] marked this conversation as resolved.
```

See [core-api/README.md](../../../opengin/core-api/README.md) for details.

3. **Start ingestion-api** (make sure CORE service is running)

For LINUX & macOS
```bash
cd opengin/ingestion-api
cp env.template .env
source .env
bal run
```

For Windows (PowerShell)
```powershell
cd opengin/ingestion-api
Copy-Item env.template.ps1 .env.ps1
. .\.env.ps1
bal run
```

See [ingestion-api/README.md](../../../opengin/ingestion-api/README.md) for details.

4. **Start read-api** (make sure CORE service is running)

For LINUX & macOS
```bash
cd opengin/read-api
cp env.template .env
source .env
bal run
```

For Windows (PowerShell)
```powershell
cd opengin/read-api
Copy-Item env.template.ps1 .env.ps1
. .\.env.ps1
bal run
```

See [read-api/README.md](../../../opengin/read-api/README.md) for details.

### Test the System

**Run E2E tests**
Expand Down
17 changes: 17 additions & 0 deletions opengin/core-api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,18 @@ source .env
# make sure to clean the dbs before running the tests
go test -v ./... -count=1 -p=1
./core-service
```

For Windows (PowerShell)
```powershell
cd core-api
Copy-Item env.template.ps1 .env.ps1
# after updating the required fields to be added to the environment
. .\.env.ps1
# make sure to clean the dbs before running the tests
go test -v ./... -count=1 -p=1
.\core-service.exe
```
Comment thread
NethmikaKekuu marked this conversation as resolved.

## Go Module Setup

Expand Down Expand Up @@ -115,10 +126,16 @@ docker run -d \

#### Validate

For LINUX & macOS
```bash
brew install grpcurl
```

For Windows, install grpcurl via Chocolatey:
```powershell
choco install grpcurl
```

```bash
grpcurl -plaintext localhost:50051 list
```
Expand Down
28 changes: 28 additions & 0 deletions opengin/core-api/env.template.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
$env:MONGO_URI=""
$env:MONGO_DB_NAME=""
$env:MONGO_COLLECTION=""

## Uncomment the following for development

# $env:NEO4J_URI=$env:NEO4J_DEVELOPMENT_DB_URI
# $env:NEO4J_USER=$env:NEO4J_DEVELOPMENT_USERNAME
# $env:NEO4J_PASSWORD=$env:NEO4J_DEVELOPMENT_PASSWORD

## If commented, uncomment the following will be used for testing

# $env:NEO4J_URI=$env:NEO4J_TESTING_DB_URI
# $env:NEO4J_USER=$env:NEO4J_TESTING_USERNAME
# $env:NEO4J_PASSWORD=$env:NEO4J_TESTING_PASSWORD

## PostgreSQL configuration

$env:POSTGRES_HOST="localhost"
$env:POSTGRES_PORT="5432"
$env:POSTGRES_USER="postgres"
$env:POSTGRES_PASSWORD="postgres"
$env:POSTGRES_DB="opengin"
$env:POSTGRES_SSL_MODE="disable"
$env:POSTGRES_TEST_DB_URI="postgresql://postgres:postgres@localhost:5432/opengin?sslmode=disable"

$env:CORE_SERVICE_HOST="localhost"
$env:CORE_SERVICE_PORT="50051"
71 changes: 45 additions & 26 deletions opengin/ingestion-api/README.md
Original file line number Diff line number Diff line change
@@ -1,56 +1,78 @@
# Ingestion API

## Generate Open API
## Prerequisites

This will generate the endpoints for the Ingestion API server using the OpenAPI specification.
⚠️ **Warning**
You should have the Ballerina version of **2201.11.0** in your device. You can download it by following link
```bash
https://ballerina.io/downloads/archived/#swan-lake-archived-versions
```

## Generate OpenAPI Service

This will generate the endpoints for the Ingestion API server using the OpenAPI specification.
The OpenAPI specification is the base for public API for Ingestion API.

> 💡 Note: Always make sure the contract has the expected endpoints and request params
> before working on the code. The generated endpoints should not be editable at all.
> before working on the code. The generated endpoints should not be editable at all.
> Maybe the only changes that can be done is adding error handlers, but request and response
> must be defined in the contract.

> must be defined in the contract.
> Note that this will create the update_api_service_copy.bal file

```bash
bal openapi -i ../contracts/rest/ingestion_api.yaml --mode service
```

## Generate GRPC Stubs
## Generate gRPC Stubs

The client stub generated here will be sending and receiving values via Grpc.
This will send requests to the corresponding CORE server endpoint.
The client stub generated here will be sending and receiving values via Grpc.
This will send requests to the corresponding CORE server endpoint.
This command will generate the types_v1_pb.bal file

```bash
bal grpc --mode client --input ../core-api/protos/types_v1.proto --output .
```

> 💡 **Note**
> At the generation make sure to remove any sample code generated to show how to use the API. Because that might add an unnecessary main file.
> At the generation make sure to remove any sample code generated to show how to use the API. Because that might add an unnecessary main file.

## Set Environmental Variables
## Environment Variables

Following are the default values you should use.
For LINUX & macOS
```bash
cd opengin/ingestion-api
cp env.template .env
# update the required fields to set the environment variables
source .env
```

For Windows (PowerShell)
```powershell
cd opengin/ingestion-api
Copy-Item env.template.ps1 .env.ps1
# after updating the required fields to be added to the environment
. .\.env.ps1
```

Example .env configuration
```bash
# For LINUX & macOS
export CORE_SERVICE_HOST=localhost
export CORE_SERVICE_PORT=50051
export INGESTION_SERVICE_HOST=localhost
export INGESTION_SERVICE_PORT=8080
```

## Development

```bash
cd ingestion-api
cp env.template .env
# update the required fields to set the environment variables
source .env
bal test
# For Windows (paste on Powershell CLI)
$env:CORE_SERVICE_HOST="localhost"
$env:CORE_SERVICE_PORT="50051"
$env:INGESTION_SERVICE_HOST="localhost"
$env:INGESTION_SERVICE_PORT="8080"
```

## Run Test
## Run Tests

Make sure the CORE server is running. (`cd design/core-api; ./core-server`)
Make sure the CORE server is running.
(LINUX & macOS: `cd opengin/core-api && ./core-service`; Windows: `cd opengin/core-api; .\core-service.exe`)
Comment thread
NethmikaKekuu marked this conversation as resolved.

```bash
# Run all tests in the current package
Expand All @@ -72,9 +94,6 @@ bal test --code-coverage
## Run Service

```bash
cd ingestion-api
cd opengin/ingestion-api
bal run
```

At the moment the port is hardcoded to 8080. This must be configurable via a config file.

11 changes: 11 additions & 0 deletions opengin/ingestion-api/env.template.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# config format 1 (default for development)
$env:CORE_SERVICE_HOST="0.0.0.0"
$env:CORE_SERVICE_PORT="50051"
$env:INGESTION_SERVICE_HOST="0.0.0.0"
$env:INGESTION_SERVICE_PORT="8080"
$env:CORE_SERVICE_URL="http://localhost:50051"

# config format 2 (default for deployment)

$env:CORE_SERVICE_URL="http://0.0.0.0:50051"
$env:INGESTION_SERVICE_URL="http://0.0.0.0:8080"
Comment thread
NethmikaKekuu marked this conversation as resolved.
Loading
Loading