QA Servers Inventory
Last updated: 2025-12-05
Summary
| Metric | Count |
|---|---|
| Total Servers | 8 |
| OTEL Ready | 2 |
| Need nginx Upgrade | 3 |
| Legacy (no OTEL) | 3 |
Server Overview
| Server | IP / Host | Hostname | OS | nginx | OTEL Module | OTEL Collector | Vector | Service Mgmt |
|---|---|---|---|---|---|---|---|---|
| qa3 | qa3.clubber.me | - | - | server/0.0 | ❌ | ❌ | - | - |
| qa4 | qa4.clubber.me | - | - | server/0.0 | ❌ | ❌ | - | - |
| qa5 | 162.55.243.21 | qa9 | Alma 9.6 | 1.20.1 | ❌ | ❌ | ❌ | daemontools + systemd |
| qa6 | 88.99.3.230 | qa6-uatfb2 | Alma 9.5 | 1.20.1 | ❌ | ❌ | ✅ systemd | daemontools + systemd |
| qa7 | 37.27.143.173 | qa7 | Alma 9.6 | 1.20.1 | ❌ | ❌ | ❌ | daemontools + systemd |
| qa8 | qa8.clubber.me | qa8 | Alma 9.3 | 1.28.0 | ✅ 0.1.2 | ✅ svc | ✅ svc | daemontools |
| qa10 | qa10.clubber.me | qa10 | Alma 9.6 | 1.28.0 | ✅ 0.1.2 | ✅ svc | ❌ | daemontools |
| dev-4 | dev-4.clubber.me | - | - | 1.16.1 | ❌ | ❌ | - | - |
OTEL Ready Servers
Only qa8 and qa10 have nginx 1.28.0 with ngx_otel_module 0.1.2 installed.
OTEL Collector Configuration Comparison
qa8 vs qa10
| Parameter | qa8 | qa10 |
|---|---|---|
| Config path | /opt/otelcol/config.yaml | /opt/otelcol-1/config.yaml |
| Receivers | otlp (grpc:4317, http:4318) | otlp (grpc:4317, http:4318) |
Exporters
| Exporter | qa8 | qa10 |
|---|---|---|
| uptrace (PROD) | ✅ uptrace-2.pne.io | ✅ uptrace-2.pne.io |
| clickstack (PROD) | ✅ clickstack-2.pne.io | ✅ clickstack-2.pne.io |
| openobserve | ❌ | ✅ openobserve-2.pne.io |
| signoz (local) | ✅ localhost:5318 | ❌ commented out |
| victorialogs (VPS) | ✅ logs.pnetest.biz:9428 | ❌ commented out |
Reliability Settings
| Setting | qa8 | qa10 |
|---|---|---|
| sending_queue | ❌ Not configured | ⚠️ Only uptrace |
| file_storage extension | ❌ | ✅ file_storage/otc |
| Prometheus metrics | ✅ :8008 | ❌ |
Pipelines
| Pipeline | qa8 | qa10 |
|---|---|---|
| traces | uptrace, clickstack, signoz | uptrace, clickstack, openobserve |
| logs | clickstack, victorialogs, signoz | uptrace, clickstack, openobserve |
Known Issues
qa8: No sending_queue configured
Traces may be lost during network issues. Recommend adding file_storage extension and sending_queue to all exporters.
qa10: Partial sending_queue
Only uptrace exporter has sending_queue. clickstack and openobserve exporters are unprotected.
qa8 Vector: VRL errors causing crash loop
Vector config has unnecessary error coalescing (to_int!(x) ?? 0). The ! operator already guarantees success, making ?? redundant. Ansible template fix ready.
qa5, qa6, qa7: nginx 1.20.1 - no OTEL support
ngx_otel_module requires nginx 1.25.3+. These servers need nginx upgrade to enable tracing.
OTEL Module Requirements
Minimum Requirements
- nginx 1.25.3 or higher
- ngx_otel_module (official nginx module)
- OTEL Collector listening on localhost:4317
Recommended Setup
- nginx 1.28.0 (latest stable)
- ngx_otel_module 0.1.2
- OTEL Collector with
sending_queue+file_storage - Vector for log aggregation
Ansible Inventory
Path: infra/ansible/inventory/qa.ini
[pne_qa_servers]
qa3 ansible_host=qa3.clubber.me ansible_user=root ansible_port=28043
qa4 ansible_host=qa4.clubber.me ansible_user=root ansible_port=28043
# hostname: qa9
qa5 ansible_host=162.55.243.21 ansible_user=root ansible_port=28043 ansible_ssh_private_key_file=~/.ssh/rwx
# hostname: qa6-uatfb2
qa6 ansible_host=88.99.3.230 ansible_user=root ansible_port=28043 ansible_ssh_private_key_file=~/.ssh/rwx
# hostname: qa7
qa7 ansible_host=37.27.143.173 ansible_user=root ansible_port=28043 ansible_ssh_private_key_file=~/.ssh/rwx
qa8 ansible_host=qa8.clubber.me ansible_user=root ansible_port=28043 ansible_ssh_private_key_file=~/.ssh/rwx
qa10 ansible_host=qa10.clubber.me ansible_user=root ansible_port=28043 ansible_ssh_private_key_file=~/.ssh/rwx
dev-4 ansible_host=dev-4.clubber.me ansible_user=root ansible_port=28043
[pne_qa_observability]
qa8
SSH Access
All servers use port 28043 and require the ~/.ssh/rwx key (except qa3, qa4, dev-4 which use default key).
# Example: connect to qa8
ssh -i ~/.ssh/rwx -p 28043 [email protected]
# Example: check nginx version on qa5
ssh -i ~/.ssh/rwx -p 28043 [email protected] "nginx -v"