Installed GGUF Models
Models in /data/models loaded on-demand via router with LRU eviction and memory guards.
| Model Identifier | Quantization | Size | RAM Fit Status | State | Actions |
|---|---|---|---|---|---|
| Scanning model directory... | |||||
Recommended CPU Architectures (MoE & Low-Active Parameters)
Optimized for high-speed AMD EPYC throughputInference Controls
API Key Management
Create, manage, and revoke client keys with SHA-256 hashed storage and sliding-window rate limits.
Connection Endpoints & Quick Reference
Copy these URLs into VS Code extensions (Cline, Continue), Cursor, or SDK clients.
Configured API Keys
0 Keys| Identifier / Description | Key Prefix | Rate Limit | Permitted Models | State | Actions |
|---|---|---|---|---|---|
| Loading keys... | |||||
Key Usage Analytics
Client Integration Snippets
API Documentation & Developer Reference
High-performance OpenAI and Anthropic compatible inference gateway powered by AMD EPYC.
1. Authentication & Base URLs
All inference endpoints are protected. Pass your generated API key using standard Bearer authentication in the HTTP header, or the Claude x-api-key header.
Authorization: Bearer sk-cpu-YOUR_SECRET_KEY
x-api-key: sk-cpu-YOUR_SECRET_KEY
2. OpenAI Compatible Endpoints
Compatible with OpenAI SDK, LangChain, LlamaIndex, LiteLLM, Continue, and Cline.
Main chat completion endpoint. Supports streaming (stream: true), system instructions, and tool calling via tools parameter.
| Field | Type | Required | Description |
|---|---|---|---|
| model | string | Yes | Model filename or alias (e.g. "gpt-4o", "qwen2.5-coder-14b-instruct-q4_k_m.gguf") |
| messages | array | Yes | Array of message objects: [{ "role": "system"|"user"|"assistant"|"tool", "content": "..." }] |
| temperature | float | No | Sampling temperature between 0.0 and 2.0 (default: 0.7) |
| stream | boolean | No | Set true for real-time Server-Sent Events (SSE) streaming chunks |
| tools | array | No | List of available tool functions with JSON Schema parameter definitions |
| max_tokens | integer | No | Maximum output tokens to generate (default: model context limit) |
curl https://ai.remlog.shop/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-cpu-YOUR_SECRET_KEY" \
-d '{
"model": "gpt-4o",
"messages": [
{"role": "system", "content": "You are a concise engineering assistant."},
{"role": "user", "content": "Explain CPU prompt caching in 2 sentences."}
],
"temperature": 0.5,
"stream": true
}'
List installed GGUF models and aliases available for routing.
Generate vector embeddings for input strings or token batches.
Legacy raw prompt completion endpoint.
3. Anthropic (Claude) Compatible Endpoints
Direct drop-in for Anthropic Python SDK, TypeScript SDK, Cline, and Continue.
Anthropic message format supporting system prompt strings, content blocks, SSE delta streams (content_block_delta), and Claude tool calling (tool_use).
import anthropic
client = anthropic.Anthropic(
base_url="https://ai.remlog.shop",
api_key="sk-cpu-YOUR_SECRET_KEY"
)
response = client.messages.create(
model="gpt-4o",
max_tokens=1024,
messages=[{"role": "user", "content": "What are Transparent Hugepages?"}]
)
print(response.content[0].text)
4. Tool Calling & Client Agent Pattern
The server handles tool planning via native Jinja templates (llama-server --jinja) and outputs standard tool calls. Clients maintain tool execution sovereignty:
tool_calls (OpenAI) or tool_use (Anthropic) response block.
role: "tool" or type: "tool_result".
5. Response Status Codes & Bounded Queue
Returns complete JSON object or streams continuous Server-Sent Events (SSE).
Sliding-window RPM limit reached. Check the Retry-After header.
Concurrency limit exceeded and queue is full. Back off and retry shortly.
Missing, invalid, or revoked API key.
Individual Core Load (18 vCPUs)
Benchmark Telemetry (llama-bench)
| Timestamp | Inference Engine | Model Identifier | Threads | Prompt Processing | Text Generation |
|---|