Tuesday, October 27, 2026

Defending osTicket and WordPress: Incident Response and Automated Blocking with CrowdSec

If you run internet-facing web applications like osTicket and WordPress, automated credential-stuffing bots will eventually find your login endpoints. To stop bursts of "Excessive login attempts" alerts hitting a login portal (/scp/login.php), investigate the server logs, confirm zero breaches occurred, and implement a multi-layered defense using UFW and CrowdSec.

Here is the full incident breakdown and the exact steps to implement this automated security setup on an Ubuntu Linux server.


1. Immediate Mitigation via Linux Firewall

When dealing with an active brute-force flood from a persistent IP address, the fastest remediation is to drop all incoming packets at the firewall before the web server expends resources processing the requests:

sudo ufw insert 1 deny from 66.232.121.113 to any
sudo ufw reload

Why insert 1 matters: Firewalls evaluate rules sequentially from top to bottom and exit on the first match. Using insert 1 places the rule at the very top of the rule chain, ensuring the traffic is dropped before any broader allow 80/tcp or allow 443/tcp web rules can accept it.


2. Post-Incident Forensics: Confirming Zero Breach

Before moving forward, verify whether the attacking IP has compromised the system by inspecting the Apache access logs and system authentication logs:

sudo grep -rn '66.232.121.113' /var/log/apache2/
sudo grep '66.232.121.113' /var/log/auth.log

The logs can confirm the following:

  • Repeated Login Loops: Every single request from the IP was exclusively cycling between GET /scp/login.php and POST /scp/login.php.
  • No Dashboard Traversal: If an authentication attempt had succeeded, osTicket would have issued a redirect to an internal page (such as /scp/index.php or /scp/tickets.php) followed by static resource requests. No internal endpoints were ever requested.
  • No Shell Access: /var/log/auth.log showed zero inbound SSH connections, confirming no external access to the underlying OS.

3. Long-Term Automated Defense with CrowdSec

Manually blocking individual IP addresses does not scale. To automate threat mitigation across osTicket, WordPress, and SSH, we installed CrowdSec along with its iptables firewall remediation bouncer.

Step A: Install CrowdSec & the Firewall Bouncer

Add the official repository and install the engine and bouncer:

# 1. Add repository and install security engine
curl -s https://install.crowdsec.net | sudo sh
sudo apt update
sudo apt install -y crowdsec

# 2. Install the iptables/firewall remediation component
sudo apt install -y crowdsec-firewall-bouncer-iptables

Step B: Enable WordPress Threat Scenarios

To guard against WordPress user enumeration, credential attacks on wp-login.php, and wp-config.php probes, install the official collection:

sudo cscli collections install crowdsecurity/wordpress
sudo systemctl reload crowdsec

Step C: Create a Custom osTicket Brute-Force Scenario

While CrowdSec includes generic HTTP protection rules, osTicket's login behavior is best served by a dedicated scenario that monitors repeated rapid POST requests targeting /login.php.

Create /etc/crowdsec/scenarios/osticket-bf.yaml:

sudo tee /etc/crowdsec/scenarios/osticket-bf.yaml > /dev/null << 'EOF'
type: leaky
name: local/osticket-bf
description: "Detect brute-force attempts on osTicket agent and client logins"
filter: "evt.Meta.log_type in ['apache2_access', 'http_access-log'] && evt.Parsed.request contains '/login.php' && evt.Parsed.verb == 'POST'"
groupby: "evt.Meta.source_ip"
capacity: 5
leakspeed: "10s"
blackhole: "5m"
labels:
  service: http
  type: bruteforce
  remediation: true
EOF

Reload the agent and verify the new scenario is registered:

sudo systemctl reload crowdsec
sudo cscli scenarios list | grep osticket

4. Operational Health & Management Commands

With this pipeline active, CrowdSec automatically monitors Apache access logs, parses incoming events, and instructs the firewall bouncer to drop malicious traffic dynamically. Useful operational commands include:

  • Check Live Parser & Bouncer Metrics: sudo cscli metrics
  • View Active Bans: sudo cscli decisions list
  • View Triggered Alerts: sudo cscli alerts list
  • Manually Ban an IP: sudo cscli decisions add --ip <IP> --duration 720h --reason "brute force"
  • Remove a False Positive Ban: sudo cscli decisions delete --ip <IP>
cfb3339690c974a7

Wednesday, October 21, 2026

Patching the Engine: Remediating Critical PHP Flaws in Production Docker Stacks

Security Advisory & Incident Response

Remediating Critical PHP Engine Flaws in Production Docker Stacks

A step-by-step technical post-mortem: runtime verification, upstream tag validation, and container cache resolution.

1. Incident Context: The Threat Vector

On October 21, 2026, coordinated emergency advisories dropped across the web infrastructure community addressing critical unauthenticated Remote Code Execution (RCE) flaws deep within core PHP request parsing and memory management routines (CVE-2026-109618 and CVE-2026-109619). The vulnerabilities reside at the engine layer, meaning typical web application firewalls and reverse proxies forwarding raw HTTP traffic cannot reliably inspect or sanitize malicious triggers.

Standard security targets required immediate updates to safe patch levels across all active branches:

  • PHP 8.2: Upgraded to 8.2.35+
  • PHP 8.3: Upgraded to 8.3.36+
  • PHP 8.4: Upgraded to 8.4.27+

2. Reverse Proxies & Ingress Filtering

A frequent initial assumption is that placing workloads behind an edge reverse proxy (such as Traefik, Nginx, or HAProxy) provides inherent layer-7 protection. In practice, unless the proxy actively implements a deep-packet inspection WAF (like Coraza/ModSecurity) with explicit signatures, it merely terminates TLS and forwards standard payloads straight downstream to the vulnerable PHP worker pools.

During a zero-day disclosure window, edge proxies should primarily be leveraged as gatekeepers—either enforcing immediate IP allowlisting or temporarily terminating public routes while downstream container images rebuild upstream.

3. Execution Workflow: Upgrading Docker Runtimes

Below is the operational runbook executed to audit, update, and harden a production container stack without relying on blind image pull assumptions.

Step 3.1: Identifying Running Runtimes

Never rely strictly on host labels or compose filenames. Execute directly against the containerized binary to verify the compiled engine version:

# Query the running container binary
docker exec -it app-core-web-production php -v

# Vulnerable baseline detected:
# PHP 8.3.25 (cli) (built: Sep  8 2025 21:19:20) (NTS)
# Zend Engine v4.3.25, OPcache v8.3.25

Step 3.2: Upstream Image Verification & Pull

Upstream registry builds lag behind official patch declarations. Pulling an image tag before automated pipelines build and push the updated binaries merely re-pulls the vulnerable layers. Checking repository tags and testing transiently ensures the fix is present before recreating the production workload:

# Pull new base layers from registry
docker pull wordpress:php8.3-apache

# Transiently spin up and confirm the underlying patch level
docker run --rm wordpress:php8.3-apache php -v

# Confirmed safe build:
# PHP 8.3.36 (cli) (built: Oct 21 2026 15:57:18) (NTS)
# Zend Engine v4.3.36, OPcache v8.3.36

Step 3.3: Configuration Hardening & Stack Recreation

To provide defense-in-depth against upload-progress heap-grooming vectors, disable progress tracking globally inside the runtime configuration (custom-php.ini):

session.upload_progress.enabled = 0

Trigger container recreation to swap the running process over to the newly verified image layer:

# Recreate production container with forced re-bind
docker compose up -d --force-recreate

# Final verification against production runtime
docker exec -it app-core-web-production php -v

4. Troubleshooting Image Cleanup: The "In-Use" Image Trap

When attempting to purge old, superseded images (often showing as <none> or legacy build hashes), Docker daemon safety checks may prevent deletion with errors stating the image is still in use.

Key Docker Behavior: Even if a container is in a Stopped state (such as staging branches, dev sandboxes, or test instances), Docker maintains an active lock on that image layer. An image cannot be pruned until every associated container is either removed or recreated against a newer image layer.

To release image references without breaking environments:

  1. For Active/Staging Workloads: Recreate the stopped or staging stacks using the freshly pulled image. As each container binds to the new image, its reference to the legacy image drops automatically.
  2. For Deprecated Environments: Remove dormant container instances explicitly:
    docker rm sandbox-app-dev staging-web-instance
    docker image prune -f

Operational Takeaways

  • Do not assume :latest tags or container rebuilds pull patched code until build timestamps and version binaries are verified inside a transient test container.
  • Layer-7 reverse proxies do not insulate backend engines from deep runtime parsing bugs without explicit WAF rules or IP-level lockdown.
  • Stopped containers maintain active image layer locks; audit staging and sandbox workloads regularly during security patch cycles.

ec6a761bc8e895a0

Thursday, October 15, 2026

Why Your Docker Containers Can't Resolve DNS: Fixing pfSense Unbound Timeouts

Fixing Periodic Docker Webhook Drops Caused by pfSense DNS Resolver (Unbound) Freezes

A comprehensive root cause analysis and step-by-step hardening guide for containerized services behind reverse proxies.

If you run containerized uptime monitors, alert dispatchers, or reverse-proxy stacks on a local server, you might run into an issue where outgoing notifications (such as Discord or Slack webhooks) suddenly stop firing. Inspecting the container logs often reveals failures like getaddrinfo ENOTFOUND or EAI_AGAIN.

A quick restart of the pfSense DNS Resolver (Unbound) usually restores connectivity immediately, but when this happens repeatedly every week or two, the root issue lies directly within how Unbound is processing queries.

Root Cause Analysis: What the Logs Show

Analyzing live Unbound logs when resolution drops reveals two hidden bottlenecks that interact to freeze the resolver service:

1. Excessive Debug Logging (Log Level ≥ 2)

What happens: The resolver writes dozens of granular trace messages (cache memory operations, module target iteration, module state transitions) per millisecond for every outbound query.

Why it breaks things: Disk and syslog I/O cannot keep up with high-frequency network queries. Unbound's single-threaded event loop blocks on write calls, causing incoming UDP DNS queries to time out and drop.

2. Iterative Root Recursion & Subquery Exhaustion

What happens: In pure resolver mode, pfSense manually walks every level of the global DNS tree (root . → TLD .com → authoritative nameservers → CNAME records) while validating multi-tiered DNSSEC cryptographic trust chains.

Why it breaks things: When any authoritative intermediate nameserver experiences packet loss or delays a response, Unbound logs event:module_event_noreply. The subqueries remain waiting in memory, worker threads pile up, and the service stops answering local LAN and Docker requests entirely.

Step-by-Step Fixes and Rationale

Step 1: Lower Logging Verbosity

  • Go to Services > DNS Resolver > Advanced Settings.
  • Set Log Level to Level 0 (or Level 1).
  • Click Save and Apply Changes.

Why we do this: Eliminates continuous disk write contention and prevents syslog thread blocking. Unbound can process thousands of lookups in memory without pausing to flush trace lines to the firewall storage.

Step 2: Enable DNS Query Forwarding & Disable Redundant DNSSEC

  • Go to Services > DNS Resolver > General Settings.
  • Check Enable Forwarding Mode under DNS Query Forwarding.
  • Uncheck DNSSEC.
  • Click Save and Apply Changes.

Why we do this: Instead of pfSense querying 5 to 10 authoritative servers across the globe for every lookup, it delegates resolution to high-speed upstream anycast servers (Cloudflare, Google). These enterprise resolvers already validate DNSSEC and maintain massive global caches, drastically cutting query response times and bypassing subquery lockups completely.

Step 3: Define Upstream Resolvers and Disable ISP Overrides

  • Go to System > General Setup.
  • Add reliable upstream IPs under DNS Server Settings (e.g., 1.1.1.1 and 8.8.8.8), explicitly selecting your WAN Gateway.
  • Uncheck DNS Server Override.
  • Click Save.

Why we do this: Explicit gateway binding ensures DNS traffic always routes directly out of the active internet link. Unchecking "DNS Server Override" prevents your local ISP from pushing slower or less reliable DNS addresses during WAN DHCP lease renewals.

Step 4: Disable Dynamic DHCP Lease Registration

  • Go to Services > DNS Resolver > General Settings.
  • Scroll to the bottom and ensure DHCP Registration (Register DHCP leases in the DNS Resolver) is unchecked.
  • Leave Static DHCP checked if you map local servers statically.
  • Click Save and Apply Changes.

Why we do this: This is the most common cause of recurring Unbound crashes in pfSense. When enabled, pfSense completely reloads the Unbound service every single time a phone, laptop, or IoT device requests or renews an IP lease. On active networks, these constant restarts corrupt the cache or deadlock the process.

Step 5: Add Container-Level DNS Fallbacks

In your Docker Compose file, explicitly define upstream resolvers under the service definition:

services:
  monitoring-service:
    image: louislam/uptime-kuma:latest
    container_name: uptime-kuma
    restart: always

    # Provide direct DNS resolvers to ensure alerting never hangs on LAN DNS
    dns:
      - 1.1.1.1
      - 8.8.8.8

    networks:
      - web

Why we do this: Establishes defense-in-depth. If your internal router is ever rebooted, updated, or temporarily unreachable, your alert monitoring containers can still resolve external APIs and dispatch critical downtime notifications without interruption.

Expected Outcome: By turning off the DHCP restart loop, offloading recursive lookups to fast upstream forwarders, and removing excessive disk logging, Unbound maintains a lean in-memory cache and stops locking up. Internal hosts and containers will consistently resolve names without requiring manual firewall intervention.