From d84a0776c6325448ed931ea7da6c237bb360dce2 Mon Sep 17 00:00:00 2001 From: jdv Date: Tue, 3 Jun 2025 17:46:53 +0200 Subject: [PATCH 01/46] init --- .../interactive_se_install/01_import_test.mdx | 13 + .../interactive_se_install/01_install.mdx | 276 ++++++++++++++++++ 2 files changed, 289 insertions(+) create mode 100644 crowdsec-docs/unversioned/user_guides/interactive_se_install/01_import_test.mdx create mode 100644 crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_import_test.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_import_test.mdx new file mode 100644 index 00000000..633d5148 --- /dev/null +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_import_test.mdx @@ -0,0 +1,13 @@ +**Ceci** est du *MDX importé.* + +Est ce qu'il est bien rendu en **HTML** ou pas ? + +--- + +## test 01 + +blabliblu + +### test 01 sub + +blobloubla \ No newline at end of file diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx new file mode 100644 index 00000000..12db36f5 --- /dev/null +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx @@ -0,0 +1,276 @@ +--- +id: i_se_install_01 +title: Security Engine Installation +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; + +import testmdximport from './01_import_test.mdx'; + +# Interractive Security Engine Installation Guide + +Welcome! This interactive guide will help you set up your CrowdSec Security Engine and validate each step to ensure proper operation. +We'll guide you through detecting and remediating malicious behavior in your services' logs, regardless of your chosen implementation (on host, Docker, Kubernetes) + +:::info +Note that to go further later you'll be able to enable Application Security and benefit from our ever growing collection of Virtual Patching rules. We'll mention the appropriate dependencies as we go along. +Lastly, note that we'll cover a setup that focuses on an autonomous security engine parsing the logs of any services from local or remote servers. +In appendix we'll cover the possibility to have distributed security engines on each servers, centralizing their alerts on a central security engine. +::: + +## Install CrowdSec + +The first step is to install the CrowdSec Security Engine somewhere you'll have access to the logs of services you want to protect. +You can choose to install it directly on the host, in a Docker container, or in a Kubernetes cluster. + +For a first experience, we recommend installing it on a host machine, as it will allow you to easily access the logs of your services and test the remediation capabilities. +But if you're comfortable with Docker it also is a great way to get started and run CrowdSec in a containerized environment. + +### Instructions + +/* We'll have to see how precise and embeded we want the various guides to be with this page + If we want full guides in here we'd better be able to embed files or pieces of files rather than copy paste + */ + + +

Here are the various hosts you can install CrowdSec on:

+ // mention guide for linux, Windows, macOS, FreeBSD, pfSense, OPNSense with links to the existing docs + + + + // mention the current guide u/getting_started/installation/docker and this page having all the env variable https://hub.docker.com/r/crowdsecurity/crowdsec +

To install CrowdSec in a Docker container, you can follow the instructions in our Docker Installation Guide.

+

For more information on the available environment variables, you can refer to the Docker Hub page.

+ + + // mention the current guide u/getting_started/installation/kubernetes and this page having all the env variable https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx +

To install CrowdSec in a Kubernetes cluster, you can follow the instructions in our Kubernetes Installation Guide.

+

For more information on the available environment variables, you can refer to the Artifact Hub page.

+ + + +### Verification + +// dump the content of the testmdximport mdx file here +// this is a test to see if the mdx import works properly +// if it does not work, we can remove this part and just have the testmdximport in the next section + + +### Troubleshooting + + + + +## Repository Installation + +The CrowdSec repository contains the latest stable version of CrowdSec and is the recommended way to install our packages. + + + +```bash +curl -s https://install.crowdsec.net | sudo sh +``` + +### Manual Repository Installation + +If you prefer to manually add the repository, you can do so by following the instructions below. + +
+ +Manual Repository Installation + + + + <> +

Begin by refreshing your package cache by running

+ sudo apt update +

If you are running Debian, install debian-archive-keyring so that official Debian repositories will be verified (Ubuntu users can skip this)

+ sudo apt install debian-archive-keyring +

Ensure the required tools (curl, gpg, apt-transport-https) are installed before proceeding:

+ sudo apt install -y curl gnupg apt-transport-https +

In order to install a deb repo, first you need to install the GPG key that used to sign repository metadata. This will change depending on whether or not your apt version is >= v.1.1. You can check this by running:

+ apt -v +

For apt version >= v1.1:

+

(Equivalent to or later than Debian/Raspbian Stretch, Ubuntu Xenial, Linux Mint Sarah, Elementary OS Loki)

+ > Create the directory to import the GPG key: + >From apt v2.4.0, `/etc/apt/keyrings/` is the designated directory for administrator imported keys. We will be using that for the following instructions, but you can replace `/etc/apt/keyrings/` with any path of your choosing. If you need to create the directory, run: + > + > mkdir -p /etc/apt/keyrings/ + > Then add the GPG key: + > + > curl -fsSL https://packagecloud.io/crowdsec/crowdsec/gpgkey | gpg --dearmor > /etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg + > Create a file named `/etc/apt/sources.list.d/crowdsec_crowdsec.list` that contains the repository configuration below. + > + > deb [signed-by=/etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg] https://packagecloud.io/crowdsec/crowdsec/any any main + deb-src [signed-by=/etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg] https://packagecloud.io/crowdsec/crowdsec/any any main + +

For apt version < v1.1:

+

(Equivalent to or older than Debian/Raspbian Jessie, Ubuntu Wily, Linux Mint Rosa, Elementary OS Freya)

+ > Add the GPG key: + > + > curl -fsSL https://packagecloud.io/crowdsec/crowdsec/gpgkey | gpg --dearmor > /etc/apt/trusted.gpg.d/crowdsec_crowdsec.gpg + > Create a file named `/etc/apt/sources.list.d/crowdsec_crowdsec.list` that contains the repository configuration below. + > + > deb https://packagecloud.io/crowdsec/crowdsec/any any main + deb-src https://packagecloud.io/crowdsec/crowdsec/any any main + +

Run this command update your local APT cache:

+ sudo apt update +

You can now install packages from your repository.

+ + + + <> +

Install pygpgme, a package which allows yum to handle gpg signatures, and a package called yum-utils which contains the tools you need for installing source RPMs.

+ sudo yum install pygpgme yum-utils +

You may need to install the EPEL repository for your system to install these packages. If you do not install pygpgme, GPG verification will not work.

+

Create a file named /etc/yum.repos.d/crowdsec_crowdsec.repo that contains the repository configuration below.

+

Make sure to replace `el` and `6` in the config below with your Linux distribution and version:

+ + [crowdsec_crowdsec] + name=crowdsec_crowdsec + baseurl=https://packagecloud.io/crowdsec/crowdsec/el/6/$basearch + repo_gpgcheck=1 + gpgcheck=1 + enabled=1 + gpgkey=https://packagecloud.io/crowdsec/crowdsec/gpgkey + https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-EDE2C695EC9A5A5C.pub.gpg + https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-C822EDD6B39954A1.pub.gpg + https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-FED78314A2468CCF.pub.gpg + sslverify=1 + sslcacert=/etc/pki/tls/certs/ca-bundle.crt + metadata_expire=3600 + + [crowdsec_crowdsec-source] + name=crowdsec_crowdsec-source + baseurl=https://packagecloud.io/crowdsec/crowdsec/el/6/SRPMS + repo_gpgcheck=1 + gpgcheck=1 + enabled=1 + gpgkey=https://packagecloud.io/crowdsec/crowdsec/gpgkey + https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-EDE2C695EC9A5A5C.pub.gpg + https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-C822EDD6B39954A1.pub.gpg + https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-FED78314A2468CCF.pub.gpg + sslverify=1 + sslcacert=/etc/pki/tls/certs/ca-bundle.crt + metadata_expire=3600 + +

Update your local yum cache by running

+ sudo yum -q makecache -y --disablerepo='*' --enablerepo='crowdsec_crowdsec' +

You can now install packages from your repository.

+ + + + +
+ +### Install Security Engine + +Once the repository is added, you can install the Security Engine via: + + + + apt install crowdsec + + + + yum install crowdsec + + + + dnf install crowdsec + + + + zypper install crowdsec + + + + opkg install crowdsec + + + + yum install crowdsec + + + + +### Install Remediation Component + +:::warning +Security Engine by itself is a detection engine, it will not block anything. You need to install a [Remediation Component](/bouncers/intro.md) to enforce decisions +::: + +For the quick start guide we will be installing the [iptables](https://en.wikipedia.org/wiki/Iptables) firewall [Remediation Component](/bouncers/intro.md). (This may not be optimal for your environment, please refer to the [Remediation Documentation](/bouncers/intro.md) for more information) + +#### IPTables + + + + +```bash +sudo apt install crowdsec-firewall-bouncer-iptables +``` + + + + +```bash +sudo yum install crowdsec-firewall-bouncer-iptables +``` + + + + + +```bash +sudo zypper install crowdsec-firewall-bouncer-iptables +``` + + + + +## Next Steps? + +Great, you now have CrowdSec installed on your system. Within the [post installation steps](/getting_started/next_steps.md) you will find the next steps to configure and optimize your installation. From 6c6cfa94aa52670bb4540bac22a7c405cb2fc998 Mon Sep 17 00:00:00 2001 From: jdv Date: Tue, 3 Jun 2025 17:47:32 +0200 Subject: [PATCH 02/46] removing junk for init --- .../interactive_se_install/01_install.mdx | 197 ------------------ 1 file changed, 197 deletions(-) diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx index 12db36f5..fdb41424 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx @@ -77,200 +77,3 @@ But if you're comfortable with Docker it also is a great way to get started and - -## Repository Installation - -The CrowdSec repository contains the latest stable version of CrowdSec and is the recommended way to install our packages. - - - -```bash -curl -s https://install.crowdsec.net | sudo sh -``` - -### Manual Repository Installation - -If you prefer to manually add the repository, you can do so by following the instructions below. - -
- -Manual Repository Installation - - - - <> -

Begin by refreshing your package cache by running

- sudo apt update -

If you are running Debian, install debian-archive-keyring so that official Debian repositories will be verified (Ubuntu users can skip this)

- sudo apt install debian-archive-keyring -

Ensure the required tools (curl, gpg, apt-transport-https) are installed before proceeding:

- sudo apt install -y curl gnupg apt-transport-https -

In order to install a deb repo, first you need to install the GPG key that used to sign repository metadata. This will change depending on whether or not your apt version is >= v.1.1. You can check this by running:

- apt -v -

For apt version >= v1.1:

-

(Equivalent to or later than Debian/Raspbian Stretch, Ubuntu Xenial, Linux Mint Sarah, Elementary OS Loki)

- > Create the directory to import the GPG key: - >From apt v2.4.0, `/etc/apt/keyrings/` is the designated directory for administrator imported keys. We will be using that for the following instructions, but you can replace `/etc/apt/keyrings/` with any path of your choosing. If you need to create the directory, run: - > - > mkdir -p /etc/apt/keyrings/ - > Then add the GPG key: - > - > curl -fsSL https://packagecloud.io/crowdsec/crowdsec/gpgkey | gpg --dearmor > /etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg - > Create a file named `/etc/apt/sources.list.d/crowdsec_crowdsec.list` that contains the repository configuration below. - > - > deb [signed-by=/etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg] https://packagecloud.io/crowdsec/crowdsec/any any main - deb-src [signed-by=/etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg] https://packagecloud.io/crowdsec/crowdsec/any any main - -

For apt version < v1.1:

-

(Equivalent to or older than Debian/Raspbian Jessie, Ubuntu Wily, Linux Mint Rosa, Elementary OS Freya)

- > Add the GPG key: - > - > curl -fsSL https://packagecloud.io/crowdsec/crowdsec/gpgkey | gpg --dearmor > /etc/apt/trusted.gpg.d/crowdsec_crowdsec.gpg - > Create a file named `/etc/apt/sources.list.d/crowdsec_crowdsec.list` that contains the repository configuration below. - > - > deb https://packagecloud.io/crowdsec/crowdsec/any any main - deb-src https://packagecloud.io/crowdsec/crowdsec/any any main - -

Run this command update your local APT cache:

- sudo apt update -

You can now install packages from your repository.

- - - - <> -

Install pygpgme, a package which allows yum to handle gpg signatures, and a package called yum-utils which contains the tools you need for installing source RPMs.

- sudo yum install pygpgme yum-utils -

You may need to install the EPEL repository for your system to install these packages. If you do not install pygpgme, GPG verification will not work.

-

Create a file named /etc/yum.repos.d/crowdsec_crowdsec.repo that contains the repository configuration below.

-

Make sure to replace `el` and `6` in the config below with your Linux distribution and version:

- - [crowdsec_crowdsec] - name=crowdsec_crowdsec - baseurl=https://packagecloud.io/crowdsec/crowdsec/el/6/$basearch - repo_gpgcheck=1 - gpgcheck=1 - enabled=1 - gpgkey=https://packagecloud.io/crowdsec/crowdsec/gpgkey - https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-EDE2C695EC9A5A5C.pub.gpg - https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-C822EDD6B39954A1.pub.gpg - https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-FED78314A2468CCF.pub.gpg - sslverify=1 - sslcacert=/etc/pki/tls/certs/ca-bundle.crt - metadata_expire=3600 - - [crowdsec_crowdsec-source] - name=crowdsec_crowdsec-source - baseurl=https://packagecloud.io/crowdsec/crowdsec/el/6/SRPMS - repo_gpgcheck=1 - gpgcheck=1 - enabled=1 - gpgkey=https://packagecloud.io/crowdsec/crowdsec/gpgkey - https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-EDE2C695EC9A5A5C.pub.gpg - https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-C822EDD6B39954A1.pub.gpg - https://packagecloud.io/crowdsec/crowdsec/gpgkey/crowdsec-crowdsec-FED78314A2468CCF.pub.gpg - sslverify=1 - sslcacert=/etc/pki/tls/certs/ca-bundle.crt - metadata_expire=3600 - -

Update your local yum cache by running

- sudo yum -q makecache -y --disablerepo='*' --enablerepo='crowdsec_crowdsec' -

You can now install packages from your repository.

- - - - -
- -### Install Security Engine - -Once the repository is added, you can install the Security Engine via: - - - - apt install crowdsec - - - - yum install crowdsec - - - - dnf install crowdsec - - - - zypper install crowdsec - - - - opkg install crowdsec - - - - yum install crowdsec - - - - -### Install Remediation Component - -:::warning -Security Engine by itself is a detection engine, it will not block anything. You need to install a [Remediation Component](/bouncers/intro.md) to enforce decisions -::: - -For the quick start guide we will be installing the [iptables](https://en.wikipedia.org/wiki/Iptables) firewall [Remediation Component](/bouncers/intro.md). (This may not be optimal for your environment, please refer to the [Remediation Documentation](/bouncers/intro.md) for more information) - -#### IPTables - - - - -```bash -sudo apt install crowdsec-firewall-bouncer-iptables -``` - - - - -```bash -sudo yum install crowdsec-firewall-bouncer-iptables -``` - - - - - -```bash -sudo zypper install crowdsec-firewall-bouncer-iptables -``` - - - - -## Next Steps? - -Great, you now have CrowdSec installed on your system. Within the [post installation steps](/getting_started/next_steps.md) you will find the next steps to configure and optimize your installation. From dd156eaf152603e0c4443b5351738eca5e4b2853 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 5 Jun 2025 14:51:37 +0200 Subject: [PATCH 03/46] 4 pages creation, page 1 full draft --- .../interactive_se_install/01_import_test.mdx | 13 --- .../interactive_se_install/01_install.mdx | 88 ++++++++++++++++--- .../02_parsers_scenarios copy.mdx | 17 ++++ .../03_acquisition copy.mdx | 17 ++++ .../interactive_se_install/04_remediation.mdx | 16 ++++ 5 files changed, 128 insertions(+), 23 deletions(-) delete mode 100644 crowdsec-docs/unversioned/user_guides/interactive_se_install/01_import_test.mdx create mode 100644 crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx create mode 100644 crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx create mode 100644 crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_import_test.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_import_test.mdx deleted file mode 100644 index 633d5148..00000000 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_import_test.mdx +++ /dev/null @@ -1,13 +0,0 @@ -**Ceci** est du *MDX importé.* - -Est ce qu'il est bien rendu en **HTML** ou pas ? - ---- - -## test 01 - -blabliblu - -### test 01 sub - -blobloubla \ No newline at end of file diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx index fdb41424..5a0d0110 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx @@ -1,14 +1,13 @@ --- -id: i_se_install_01 +id: se_install_01 title: Security Engine Installation +pagination_next: /u/user_guides/interactive_se_install/se_install_02 --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; -import testmdximport from './01_import_test.mdx'; - # Interractive Security Engine Installation Guide Welcome! This interactive guide will help you set up your CrowdSec Security Engine and validate each step to ensure proper operation. @@ -68,12 +67,81 @@ But if you're comfortable with Docker it also is a great way to get started and ### Verification -// dump the content of the testmdximport mdx file here -// this is a test to see if the mdx import works properly -// if it does not work, we can remove this part and just have the testmdximport in the next section - +Let's check that CrowdSec is running and able to retrieve the community blocklist ! + +#### CrowdSec installation health + +[] Check that the CrowdSec service is running +```bash +systemctl status crowdsec +``` +- You should see the service status as "active (running)". +- The port configuration can be setup in config.yaml file or by setting the environment variables depending on your implementation + +[] Check that your Local API (LAPI) is properly running: +```bash +sudo cscli machines list +``` +- You should see a list of machines, including the one you just installed CrowdSec on. +- For an autonomous install the Security Engine is running and connecting to it's own LAPI +- You should see a recent last heartbeat and a checkmark in status. + +[] Check the metrics can be queried (optionnal for cscli metrics and prometheus): +```bash +sudo cscli metrics +``` +- CrowdSec serves the metrics via a prometheus endpoint, check that it's able to run properly +- You should see various tables, most still empty except the Local API Machines Metrics + +[] Setup hub-update // ?@seb case we need this ? +- ... + +#### CrowdSec connectivity health + +[] Check your security engine can connect to the Central API +```bash +sudo cscli capi status +``` +- You should see: "You can successfully interact with Central API (CAPI)" +- Optionally additional status + - Sharing signals is enabled //+link to doc where to turn this on/off ? + - Pulling community blocklist is enabled //+link to doc where to turn this on/off ? + - Pulling blocklists from the console is enabled //+link to doc where to turn this on/off ? + +#### Enroll your Security Engine into CrowdSec Console +For advanced monitoring and trouble shooting the CrowdSec Console is a great tool to visualize your Security Engine's activity and alerts. + +[] Enroll into the console +- [link to doc] +- You'll see a confirmation pop up in the console for enrollment if not you might have conectivity issue to the central API [link to troubleshooting section] +- You'll see the last heartbeat and the status of your Security Engine in the console +- You'll be able to check various configurations for the upcoming steps of the installation +- You'll be warned when a new version of CrowdSec is available ### Troubleshooting - - - +
+ There could be ports conflicts with other services + + The Local API is running on port 8080 by default, and the Metrics server is running on port 6060 by default. + If you have other services running on these ports, you can change the ports in the configuration file `/etc/crowdsec/config.yaml` or by setting the environment variables `CROWDSEC_API_PORT` and `CROWDSEC_METRICS_PORT` when running CrowdSec in a container. +
+
+ No connectivity to Central API + + Check that you have access to internet at least api.crowdsec.net + Check that your online api credentials exist /etc/crowdsec/config/online_api_credentials.yaml (default path) + Eventually reset them with the following command sudo cscli capi register +
+
+ Enrollment in Console not working + + Make sure you are looking in the proper organization in the console: the enrollment key is linked to your organization. + If you have multiple organizations, you can switch using the organization selector in the top left corner of the console. + If you already checked that there are no connectivity issues, you can try to re-enroll using the --overwrite flag, effectively forcing the engine to link to your organization. +
+
+ Can't see the latest version of the package + + Update your repository and install again. + If latest version not available, Might depend on the plateform you're installing it on (be patient, it will come) +
\ No newline at end of file diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx new file mode 100644 index 00000000..bf9d8810 --- /dev/null +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx @@ -0,0 +1,17 @@ +--- +id: se_install_02 +title: Parsers and Scenarios Selection +pagination_prev: /u/user_guides/interactive_se_install/se_install_01 +pagination_next: /u/user_guides/interactive_se_install/se_install_03 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; + +# Selection of Parsers and Scenarios + +blabla + +## Choosing what youneed to protect your services + diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx new file mode 100644 index 00000000..6fe14618 --- /dev/null +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx @@ -0,0 +1,17 @@ +--- +id: se_install_03 +title: Parsers and Scenarios Selection +pagination_prev: /u/user_guides/interactive_se_install/se_install_02 +pagination_next: /u/user_guides/interactive_se_install/se_install_04 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; + +# Acquisition + +blabla + +## Setting up acquisition datasources for detection + diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx new file mode 100644 index 00000000..9f79775d --- /dev/null +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx @@ -0,0 +1,16 @@ +--- +id: se_install_04 +title: Parsers and Scenarios Selection +pagination_prev: /u/user_guides/interactive_se_install/se_install_03 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; + +# Remediation + +blabla + +## Choosing and testing a remediation component + From 51240a548c2f04fa5f632c92a697916c2dd082b8 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 5 Jun 2025 15:17:10 +0200 Subject: [PATCH 04/46] fix prev nav --- .../user_guides/interactive_se_install/01_install.mdx | 2 +- .../interactive_se_install/02_parsers_scenarios copy.mdx | 6 +++--- .../interactive_se_install/03_acquisition copy.mdx | 4 ++-- .../user_guides/interactive_se_install/04_remediation.mdx | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx index 5a0d0110..5f5d1d21 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx @@ -1,7 +1,7 @@ --- id: se_install_01 title: Security Engine Installation -pagination_next: /u/user_guides/interactive_se_install/se_install_02 +pagination_next: user_guides/interactive_se_install/se_install_02 --- import Tabs from '@theme/Tabs'; diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx index bf9d8810..58842611 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx @@ -1,8 +1,8 @@ --- id: se_install_02 title: Parsers and Scenarios Selection -pagination_prev: /u/user_guides/interactive_se_install/se_install_01 -pagination_next: /u/user_guides/interactive_se_install/se_install_03 +pagination_prev: user_guides/interactive_se_install/se_install_01 +pagination_next: user_guides/interactive_se_install/se_install_03 --- import Tabs from '@theme/Tabs'; @@ -13,5 +13,5 @@ import CodeBlock from '@theme/CodeBlock'; blabla -## Choosing what youneed to protect your services +## Choosing what you need to protect your services diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx index 6fe14618..1d972848 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx @@ -1,8 +1,8 @@ --- id: se_install_03 title: Parsers and Scenarios Selection -pagination_prev: /u/user_guides/interactive_se_install/se_install_02 -pagination_next: /u/user_guides/interactive_se_install/se_install_04 +pagination_prev: user_guides/interactive_se_install/se_install_02 +pagination_next: user_guides/interactive_se_install/se_install_04 --- import Tabs from '@theme/Tabs'; diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx index 9f79775d..cf8d9f49 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx @@ -1,7 +1,7 @@ --- id: se_install_04 title: Parsers and Scenarios Selection -pagination_prev: /u/user_guides/interactive_se_install/se_install_03 +pagination_prev: user_guides/interactive_se_install/se_install_03 --- import Tabs from '@theme/Tabs'; From 5f20433a8e222d4a5a29fd3f8a7bc90121cc18e1 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 5 Jun 2025 15:29:59 +0200 Subject: [PATCH 05/46] nano visual update --- .../interactive_se_install/01_install.mdx | 28 +++++++++++++++---- 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx index 5f5d1d21..1ff0c455 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx @@ -71,14 +71,19 @@ Let's check that CrowdSec is running and able to retrieve the community blocklis #### CrowdSec installation health -[] Check that the CrowdSec service is running +:::info +[ ] Check that the CrowdSec service is running +::: + ```bash systemctl status crowdsec ``` - You should see the service status as "active (running)". - The port configuration can be setup in config.yaml file or by setting the environment variables depending on your implementation -[] Check that your Local API (LAPI) is properly running: +:::info +[ ] Check that your Local API (LAPI) is properly running: +::: ```bash sudo cscli machines list ``` @@ -86,19 +91,28 @@ sudo cscli machines list - For an autonomous install the Security Engine is running and connecting to it's own LAPI - You should see a recent last heartbeat and a checkmark in status. -[] Check the metrics can be queried (optionnal for cscli metrics and prometheus): +:::info +[ ] Check the metrics can be queried (optionnal for cscli metrics and prometheus): +::: + ```bash sudo cscli metrics ``` - CrowdSec serves the metrics via a prometheus endpoint, check that it's able to run properly - You should see various tables, most still empty except the Local API Machines Metrics -[] Setup hub-update // ?@seb case we need this ? +:::info +[ ] Setup hub-update // ?@seb case we need this ? +::: + - ... #### CrowdSec connectivity health -[] Check your security engine can connect to the Central API +:::info +[ ] Check your security engine can connect to the Central API +::: + ```bash sudo cscli capi status ``` @@ -111,7 +125,9 @@ sudo cscli capi status #### Enroll your Security Engine into CrowdSec Console For advanced monitoring and trouble shooting the CrowdSec Console is a great tool to visualize your Security Engine's activity and alerts. -[] Enroll into the console +:::info +[ ] Enroll into the console +::: - [link to doc] - You'll see a confirmation pop up in the console for enrollment if not you might have conectivity issue to the central API [link to troubleshooting section] - You'll see the last heartbeat and the status of your Security Engine in the console From 85d0f2dfecb3d4467b1bb16a939b6d0fedb4785a Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 5 Jun 2025 16:25:51 +0200 Subject: [PATCH 06/46] nano visual update test --- .../interactive_se_install/01_install.mdx | 26 ++++++------------- 1 file changed, 8 insertions(+), 18 deletions(-) diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx index 1ff0c455..f91ac071 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx @@ -71,9 +71,7 @@ Let's check that CrowdSec is running and able to retrieve the community blocklis #### CrowdSec installation health -:::info -[ ] Check that the CrowdSec service is running -::: +> [ ] Check that the CrowdSec service is running ```bash systemctl status crowdsec @@ -81,9 +79,8 @@ systemctl status crowdsec - You should see the service status as "active (running)". - The port configuration can be setup in config.yaml file or by setting the environment variables depending on your implementation -:::info -[ ] Check that your Local API (LAPI) is properly running: -::: +> [ ] Check that your Local API (LAPI) is properly running: + ```bash sudo cscli machines list ``` @@ -91,9 +88,7 @@ sudo cscli machines list - For an autonomous install the Security Engine is running and connecting to it's own LAPI - You should see a recent last heartbeat and a checkmark in status. -:::info -[ ] Check the metrics can be queried (optionnal for cscli metrics and prometheus): -::: +> [ ] Check the metrics can be queried (optionnal for cscli metrics and prometheus): ```bash sudo cscli metrics @@ -101,17 +96,13 @@ sudo cscli metrics - CrowdSec serves the metrics via a prometheus endpoint, check that it's able to run properly - You should see various tables, most still empty except the Local API Machines Metrics -:::info -[ ] Setup hub-update // ?@seb case we need this ? -::: +> [ ] Setup hub-update // ?@seb case we need this ? - ... #### CrowdSec connectivity health -:::info -[ ] Check your security engine can connect to the Central API -::: +> [ ] Check your security engine can connect to the Central API ```bash sudo cscli capi status @@ -125,9 +116,8 @@ sudo cscli capi status #### Enroll your Security Engine into CrowdSec Console For advanced monitoring and trouble shooting the CrowdSec Console is a great tool to visualize your Security Engine's activity and alerts. -:::info -[ ] Enroll into the console -::: +> [ ] Enroll into the console + - [link to doc] - You'll see a confirmation pop up in the console for enrollment if not you might have conectivity issue to the central API [link to troubleshooting section] - You'll see the last heartbeat and the status of your Security Engine in the console From decb4c36585e1dc9589d1b889798bf1922c6f8e9 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 5 Jun 2025 17:27:44 +0200 Subject: [PATCH 07/46] section 2 firs draft --- .../02_parsers_scenarios copy.mdx | 59 +++++++++++++++++++ .../03_acquisition copy.mdx | 21 +++++++ .../interactive_se_install/04_remediation.mdx | 20 +++++++ 3 files changed, 100 insertions(+) diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx index 58842611..b76be598 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx @@ -15,3 +15,62 @@ blabla ## Choosing what you need to protect your services +To detect malicious behaviors targeting your services the (CrowdSec) Security Engine will need t be able to parse your services' logs and apply detection rules on them AKA **Scenarios**. +Our parsers and Scenarios are available on the [CrowdSec Hub](https://hub.crowdsec.net/), and are regrouped into **Collections** to make it more convenient to install as a whole. +Let's see what parsers and Scenarios you need to install to protect your services, and how to install them. + +### Instructions + +A typical use case would be to want to protect a web server, having an open ssh access. +You'd want parsers for your webserver logs (e.g. Nginx, Apache) and for your SSH logs, as well as the corresponding Scenarios to detect malicious behaviors. +In the case of NGINX being your reverse proxy you would want the NGINX parser AND all the scenarios related to protecting HTTP: +The [nginx collection](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx) would be one you want to install. +- It embeds the NGINX parser, as well as the Base HTTP scenarios collection and an additionnal NGINX specific scenario triggerring on nginx request limit exceeded. +- You can install it with the following command: +```bash +crowdsec collections install crowdsecurity/nginx +``` + +To protect your SSH accessed Linux server you would want the [linux collection](https://app.crowdsec.net/hub/author/crowdsecurity/collections/linux) which includes the Linux parser and a set of scenarios to detect malicious SSH behaviors. +- You can install it with the following command: +```bash +crowdsec collections install crowdsecurity/linux +``` + +One way you cna proceed for your own usecase is: +1. look for a logparser specific for your service in the [parser's section of the hub](https://app.crowdsec.net/hub/log-parsers) +2. Once you have found the parser you need, check if there is a collection with the same name. If so, chose the colleciton it will be pre-packed with reommanded scenarios. +3. You can browse other collections too or individual scenarios you might want to install. + - Note that you're also able to [create your own scenarios if needed](https://doc.crowdsec.net/docs/next/log_processor/parsers/intro). +4. Run the install you can find in the corresponding hub page +5. Keep the hub page in mind there are usefull instrucitons to the next part of this guide. + +### Verification + +Let's check the installation of the parsers and scenarios was successfull. +After installing them you should restart the CrowdSec service to ensure the new parsers and scenarios are loaded: +```bash +sudo systemctl restart crowdsec +``` + +#### List of installed parsers and scenarios + +> [ ] Check Collections, parsers and scenarios are installed +```bash +cscli collections list +cscli parsers list +cscli scenarios list +``` +- You'll be able to verify that the parsers and scenarios you installed are listed in the output of these commands. + +> [ ] Check scenarios are showing up in the console +When clicking on the "Scenarios" part of your [Security engine tile in the CrowdSec console](https://app.crowdsec.net/security-engines), you should see the scenarios you installed listed there. +The tile itself should show you the count of scenarios that are installed on your Security Engine. + +### Troubleshooting + +
+ Missing Scenarios + + @seb @laurence any reason why a scenario or colleciton would not install ? folder issue ? config ? +
\ No newline at end of file diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx index 1d972848..e207d4bc 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx @@ -15,3 +15,24 @@ blabla ## Setting up acquisition datasources for detection + +### Instructions +... + +### Verification + +Let's check ... + +#### CrowdSec installation health + +> [ ] Check ... + +```bash +... +``` +- You should see ... +- ... + + + +### Troubleshooting diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx index cf8d9f49..750012fd 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx @@ -14,3 +14,23 @@ blabla ## Choosing and testing a remediation component +### Instructions +... + +### Verification + +Let's check ... + +#### CrowdSec installation health + +> [ ] Check ... + +```bash +... +``` +- You should see ... +- ... + + + +### Troubleshooting From 7fcfcd5f8da3d34e8d777e851120e93de70f841f Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 5 Jun 2025 18:28:12 +0200 Subject: [PATCH 08/46] section 3 first draft --- .../03_acquisition copy.mdx | 56 +++++++++++++++---- 1 file changed, 46 insertions(+), 10 deletions(-) diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx index e207d4bc..58bda31e 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx @@ -11,28 +11,64 @@ import CodeBlock from '@theme/CodeBlock'; # Acquisition -blabla +Acquisition setup indicates how to retrieve the logs (file, syslog,...) and what parser to use for those logs. +The configuration for all acquisitions is located in /etc/crowdsec/acquis.yaml, you can choose to split and store them in /etc/crowdsec/acquis.d/.yaml (both method can coexist) ## Setting up acquisition datasources for detection +TODO: explain how to infer the type give link to collection of prefilled with nicely named placeholders example for all datasource. give 2 examples for each: one minimal and one with all optional fields (+how to use wildcard and exclude for files for example) - can we make a gtp prompt ? ### Instructions -... +... per datasource ? ### Verification -Let's check ... +Let's check that parsing is working for all installed parsers and logs. -#### CrowdSec installation health - -> [ ] Check ... +#### Check acquisition: all necessary logs are read and parsed properly +> [ ] Files are properly read and parsed +- Use your services normally: navigate on your website, log in via ssh ... +- Check the metrics ```bash -... +sudo cscli metrics ``` -- You should see ... -- ... - +Alternatively to select only the metrics we're interrested in here: +```bash +sudo cscli metrics show acquisition parsers +``` +- You should see names of the log files you configured in the acquisition section, and the number of lines parsed for each of them. + - The number of "Lines parsed" should be non-zero for each of the files you configured in the acquisition section. and for most cases should be equal to the number of "lines read". +- The parsers metrics show you what parsers were successfully used. Look for the name of the parsers you installed in the previous step +#### Run on existing logs if you have some +> [ ] Check alerts in past logs +- Most services open to the internet for more than a few days will have some malicious activity in their logs. +- You can run the Security Engine on existing logs to check if it detects any malicious activity. +- ... blabla we have DSN commands in datasources docs. +- Try... +- You will probably see alerts raised when doing this command: +```bash +sudo cscli alerts list +``` +- The alerts will also show up in the console within a few minutes if you enrolled your Security Engine. + ### Troubleshooting +If you never see any alerts: However unlikely it's possible that you're not being hit by attacks but first lets check what could be wrong. + +
+ IPs in your logs not properly X-forwarded for + + - Check that the logs you are reading have the real Source IP and not the one of one of your proxies or load balancers. + - Local/Private IPs are whitelisted by default, so log with non x-forwarded IPs wont trigger alerts. //more details here.. link ... + - If you have a reverse proxy, make sure to use the `X-Forwarded-For` header in your logs. + - If you have a load balancer, make sure to use the `X-Real-IP` header in your logs. +
+
+ You are using custom log formats + + The parsers we provide are made to parse default log formats of the supported services. + If you are using custom log formats, you will need to create your own parsers or modify the existing ones to match your log format. + - You can find more information on how to create your own parsers in the [CrowdSec documentation](https://doc.crowdsec.net/docs/next/log_processor/parsers/intro). +
\ No newline at end of file From bd85ca29564d272eaffcf6d08ceaf5dd06603093 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 5 Jun 2025 18:34:19 +0200 Subject: [PATCH 09/46] section 4 first draft --- .../interactive_se_install/04_remediation.mdx | 35 ++++++++++++++----- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx index 750012fd..666fe03f 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx @@ -10,27 +10,44 @@ import CodeBlock from '@theme/CodeBlock'; # Remediation -blabla +Now that the detection is setup, lets remediate those bad IPs. You can apply a remediation in many places. +The job of a remediation component (aka RC or bouncer) is to keep the list of IPs up to date within the context you want to use: from the firewall of your host iptables/nftables to your edge appliances by exposing the list of IPs to it (mirror bouncer) through AWS WAF or Cloudflare... See full list of remediation components here[link] +This list of IPs will be comprised of decisions taken by the security engine (based on scenarios), IPs from blocklists your Security Engine is subscribed to (by default the community blocklist), and manual additions you may do via various methods. ## Choosing and testing a remediation component +// Explain various ways and advantages (push to be able to appsec and has metrics) + ### Instructions ... +//Help connect to SE +A remediation component can get its information from the security engine via a URL+TOKEN (some support Mutual TLS authentication) +If installed on a host with a running security engine, they usually automatically pair (you can check with cscli bouncers list) to change the pairing, follow the steps described in the documentation + ### Verification Let's check ... -#### CrowdSec installation health - -> [ ] Check ... - +#### Bouncer connectivity +> [ ] Check that the bouncer is connected to the Security Engine +Check that the bouncer is registered properly and active (in cscli and in the console) ```bash -... +cscli bouncers list ``` -- You should see ... -- ... - +- You should see the bouncer you just installed in the list, with a recent heartbeat and a valid tick mark +> [ ] Check that the bouncing is effective +- Have them add a decision for the IP of our canary automate (or use a free blocklist we use for the guide with 10 most aggressive IPs and one IP of our test automate canary) +- Have them provide a URL that supposedly is protected +- Have the canary try to reach and return a response +- ??? How do we do this if we don't have a canary? without asking them to ban themselves +// find an alternative while we don't have a canary system @thib @seb @laurence ### Troubleshooting + +
+ Docker communication with bouncer + + Docker bouncer: warning docker chain for example with firewall bouncer for docker user chain... ask for more precise phrasing @laurence +
\ No newline at end of file From c18d9f211f40b0216ce3e02e114916e3f643058d Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 5 Jun 2025 18:41:17 +0200 Subject: [PATCH 10/46] fixed typo in section 3 --- .../user_guides/interactive_se_install/03_acquisition copy.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx index 58bda31e..236858a4 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx @@ -12,7 +12,7 @@ import CodeBlock from '@theme/CodeBlock'; # Acquisition Acquisition setup indicates how to retrieve the logs (file, syslog,...) and what parser to use for those logs. -The configuration for all acquisitions is located in /etc/crowdsec/acquis.yaml, you can choose to split and store them in /etc/crowdsec/acquis.d/.yaml (both method can coexist) +The configuration for all acquisitions is located in /etc/crowdsec/acquis.yaml, you can choose to split and store them in /etc/crowdsec/acquis.d/\.yaml (both method can coexist) ## Setting up acquisition datasources for detection From 67f8e19914ffd6893adb891c85427d07c2e0f88c Mon Sep 17 00:00:00 2001 From: jdv Date: Tue, 10 Jun 2025 15:13:26 +0200 Subject: [PATCH 11/46] test interactiveCheckboxes --- .../src/components/InteractiveCheckbox.js | 160 ++++++++++++++++++ .../interactive_se_install/01_install.mdx | 29 ++-- .../02_parsers_scenarios copy.mdx | 2 +- .../03_acquisition copy.mdx | 2 +- .../interactive_se_install/04_remediation.mdx | 2 +- 5 files changed, 178 insertions(+), 17 deletions(-) create mode 100644 crowdsec-docs/src/components/InteractiveCheckbox.js diff --git a/crowdsec-docs/src/components/InteractiveCheckbox.js b/crowdsec-docs/src/components/InteractiveCheckbox.js new file mode 100644 index 00000000..102ee851 --- /dev/null +++ b/crowdsec-docs/src/components/InteractiveCheckbox.js @@ -0,0 +1,160 @@ +import React, { useState, useEffect, createContext, useContext } from 'react'; + +// Global context to manage checkbox states across all instances +const CheckboxContext = createContext(); + +// Provider component to wrap your MDX content +export const CheckboxProvider = ({ children }) => { + const [checkboxStates, setCheckboxStates] = useState({}); + const [manualChecks, setManualChecks] = useState({}); + + const updateCheckbox = (id, isManuallyChecked) => { + setManualChecks(prev => ({ + ...prev, + [id]: isManuallyChecked + })); + }; + + const isCheckboxChecked = (id, references = []) => { + // Manual check has priority + if (manualChecks[id]) { + return { checked: true, type: 'manual' }; + } + + // Check if all references are checked (either manually or through their references) + if (references.length > 0) { + const allReferencesChecked = references.every(refId => { + if (manualChecks[refId]) return true; + // Recursively check if reference has its own references + return checkboxStates[refId]?.checked; + }); + + if (allReferencesChecked) { + return { checked: true, type: 'reference' }; + } + } + + return { checked: false, type: 'none' }; + }; + + useEffect(() => { + // Update all checkbox states when manual checks change + const newStates = {}; + Object.keys(checkboxStates).forEach(id => { + const checkbox = checkboxStates[id]; + newStates[id] = { + ...checkbox, + ...isCheckboxChecked(id, checkbox.references) + }; + }); + setCheckboxStates(newStates); + }, [manualChecks]); + + const registerCheckbox = (id, references = []) => { + setCheckboxStates(prev => ({ + ...prev, + [id]: { + references, + ...isCheckboxChecked(id, references) + } + })); + }; + + return ( + + {children} + + ); +}; + +// The main checkbox component +export const InteractiveCheckbox = ({ + id, + references = [], + label = '', + className = '' +}) => { + const { + checkboxStates, + manualChecks, + updateCheckbox, + registerCheckbox, + isCheckboxChecked + } = useContext(CheckboxContext); + + useEffect(() => { + registerCheckbox(id, references); + }, [id, references]); + + const handleClick = () => { + updateCheckbox(id, !manualChecks[id]); + }; + + const checkboxState = isCheckboxChecked(id, references); + const isChecked = checkboxState.checked; + const checkType = checkboxState.type; + + // Determine the color based on check type + const getCheckmarkColor = () => { + if (checkType === 'manual') return '#22c55e'; // green + if (checkType === 'reference') return '#3b82f6'; // blue + return '#d1d5db'; // gray (unchecked) + }; + + const getBackgroundColor = () => { + if (isChecked) return getCheckmarkColor(); + return 'transparent'; + }; + + return ( +
+
+ {isChecked && ( + + + + )} +
+ {label && ( + + )} +
+ ); +}; + +// Export both components as default for easier importing +export default InteractiveCheckbox; \ No newline at end of file diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx index f91ac071..1072356f 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx @@ -7,8 +7,9 @@ pagination_next: user_guides/interactive_se_install/se_install_02 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import { InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; -# Interractive Security Engine Installation Guide +# Interactive Security Engine Installation Guide Welcome! This interactive guide will help you set up your CrowdSec Security Engine and validate each step to ensure proper operation. We'll guide you through detecting and remediating malicious behavior in your services' logs, regardless of your chosen implementation (on host, Docker, Kubernetes) @@ -69,9 +70,20 @@ But if you're comfortable with Docker it also is a great way to get started and Let's check that CrowdSec is running and able to retrieve the community blocklist ! +#### Enroll your Security Engine into CrowdSec Console +For advanced monitoring and trouble shooting the CrowdSec Console is a great tool to visualize your Security Engine's activity and alerts. + + + +- [link to doc] +- You'll see a confirmation pop up in the console for enrollment if not you might have conectivity issue to the central API [link to troubleshooting section] +- You'll see the last heartbeat and the status of your Security Engine in the console +- You'll be able to check various configurations for the upcoming steps of the installation +- You'll be warned when a new version of CrowdSec is available + #### CrowdSec installation health -> [ ] Check that the CrowdSec service is running + ```bash systemctl status crowdsec @@ -113,18 +125,7 @@ sudo cscli capi status - Pulling community blocklist is enabled //+link to doc where to turn this on/off ? - Pulling blocklists from the console is enabled //+link to doc where to turn this on/off ? -#### Enroll your Security Engine into CrowdSec Console -For advanced monitoring and trouble shooting the CrowdSec Console is a great tool to visualize your Security Engine's activity and alerts. - -> [ ] Enroll into the console - -- [link to doc] -- You'll see a confirmation pop up in the console for enrollment if not you might have conectivity issue to the central API [link to troubleshooting section] -- You'll see the last heartbeat and the status of your Security Engine in the console -- You'll be able to check various configurations for the upcoming steps of the installation -- You'll be warned when a new version of CrowdSec is available - -### Troubleshooting +### 🚨 Troubleshooting
There could be ports conflicts with other services diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx index b76be598..4f6601a8 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx @@ -67,7 +67,7 @@ cscli scenarios list When clicking on the "Scenarios" part of your [Security engine tile in the CrowdSec console](https://app.crowdsec.net/security-engines), you should see the scenarios you installed listed there. The tile itself should show you the count of scenarios that are installed on your Security Engine. -### Troubleshooting +### 🚨 Troubleshooting
Missing Scenarios diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx index 236858a4..eebccbbe 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx @@ -54,7 +54,7 @@ sudo cscli alerts list ``` - The alerts will also show up in the console within a few minutes if you enrolled your Security Engine. -### Troubleshooting +### 🚨 Troubleshooting If you never see any alerts: However unlikely it's possible that you're not being hit by attacks but first lets check what could be wrong.
diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx index 666fe03f..f3e20565 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx @@ -44,7 +44,7 @@ cscli bouncers list - ??? How do we do this if we don't have a canary? without asking them to ban themselves // find an alternative while we don't have a canary system @thib @seb @laurence -### Troubleshooting +### 🚨 Troubleshooting
Docker communication with bouncer From 24280d32ad0e1d3797f33b0ab8341f65d0a73c34 Mon Sep 17 00:00:00 2001 From: jdv Date: Tue, 10 Jun 2025 16:10:25 +0200 Subject: [PATCH 12/46] fixing checkboxes --- .../src/components/InteractiveCheckbox.js | 40 ++++++++++++++++--- .../interactive_se_install/01_install.mdx | 6 +-- 2 files changed, 38 insertions(+), 8 deletions(-) diff --git a/crowdsec-docs/src/components/InteractiveCheckbox.js b/crowdsec-docs/src/components/InteractiveCheckbox.js index 102ee851..bfdf1b54 100644 --- a/crowdsec-docs/src/components/InteractiveCheckbox.js +++ b/crowdsec-docs/src/components/InteractiveCheckbox.js @@ -80,23 +80,53 @@ export const InteractiveCheckbox = ({ label = '', className = '' }) => { + const context = useContext(CheckboxContext); + + // Fallback if context is not available (useful for development/SSR) + if (!context) { + console.warn('InteractiveCheckbox must be used within a CheckboxProvider'); + return ( +
+
+ {label && {label}} +
+ ); + } + const { checkboxStates, manualChecks, updateCheckbox, registerCheckbox, isCheckboxChecked - } = useContext(CheckboxContext); + } = context; useEffect(() => { - registerCheckbox(id, references); - }, [id, references]); + if (registerCheckbox) { + registerCheckbox(id, references); + } + }, [id, references, registerCheckbox]); const handleClick = () => { - updateCheckbox(id, !manualChecks[id]); + if (updateCheckbox) { + updateCheckbox(id, !manualChecks[id]); + } }; - const checkboxState = isCheckboxChecked(id, references); + const checkboxState = isCheckboxChecked ? isCheckboxChecked(id, references) : { checked: false, type: 'none' }; const isChecked = checkboxState.checked; const checkType = checkboxState.type; diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx index 1072356f..a824729c 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx @@ -7,7 +7,7 @@ pagination_next: user_guides/interactive_se_install/se_install_02 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; -import { InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; +import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; # Interactive Security Engine Installation Guide @@ -72,7 +72,7 @@ Let's check that CrowdSec is running and able to retrieve the community blocklis #### Enroll your Security Engine into CrowdSec Console For advanced monitoring and trouble shooting the CrowdSec Console is a great tool to visualize your Security Engine's activity and alerts. - + - [link to doc] @@ -84,7 +84,7 @@ For advanced monitoring and trouble shooting the CrowdSec Console is a great too #### CrowdSec installation health - + ```bash systemctl status crowdsec ``` From ed12c11a16808b1bf847e2fff3186a46349206e7 Mon Sep 17 00:00:00 2001 From: jdv Date: Tue, 10 Jun 2025 17:22:11 +0200 Subject: [PATCH 13/46] reverting change on checkbox for now --- .../src/components/InteractiveCheckbox.js | 58 +++++++++++++------ .../interactive_se_install/01_install.mdx | 9 ++- 2 files changed, 44 insertions(+), 23 deletions(-) diff --git a/crowdsec-docs/src/components/InteractiveCheckbox.js b/crowdsec-docs/src/components/InteractiveCheckbox.js index bfdf1b54..f11a5e3c 100644 --- a/crowdsec-docs/src/components/InteractiveCheckbox.js +++ b/crowdsec-docs/src/components/InteractiveCheckbox.js @@ -3,8 +3,8 @@ import React, { useState, useEffect, createContext, useContext } from 'react'; // Global context to manage checkbox states across all instances const CheckboxContext = createContext(); -// Provider component to wrap your MDX content -export const CheckboxProvider = ({ children }) => { +// Hook to create a standalone checkbox manager +export const useCheckboxManager = () => { const [checkboxStates, setCheckboxStates] = useState({}); const [manualChecks, setManualChecks] = useState({}); @@ -60,14 +60,21 @@ export const CheckboxProvider = ({ children }) => { })); }; + return { + checkboxStates, + manualChecks, + updateCheckbox, + registerCheckbox, + isCheckboxChecked + }; +}; + +// Provider component to wrap your MDX content (optional) +export const CheckboxProvider = ({ children }) => { + const checkboxManager = useCheckboxManager(); + return ( - + {children} ); @@ -78,13 +85,17 @@ export const InteractiveCheckbox = ({ id, references = [], label = '', - className = '' + className = '', + manager = null // Accept a manager prop }) => { - const context = useContext(CheckboxContext); + const contextManager = useContext(CheckboxContext); - // Fallback if context is not available (useful for development/SSR) - if (!context) { - console.warn('InteractiveCheckbox must be used within a CheckboxProvider'); + // Use the passed manager or fall back to context, then to a warning fallback + const checkboxManager = manager || contextManager; + + // Fallback if no manager is available + if (!checkboxManager) { + console.warn('InteractiveCheckbox must either be used within a CheckboxProvider or have a manager prop passed to it'); return (
{ if (registerCheckbox) { @@ -143,7 +154,10 @@ export const InteractiveCheckbox = ({ }; return ( -
+
{isChecked && ( @@ -177,7 +192,14 @@ export const InteractiveCheckbox = ({ {label && ( diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx index a824729c..e417dbd2 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx @@ -7,7 +7,8 @@ pagination_next: user_guides/interactive_se_install/se_install_02 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; -import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; +import { useCheckboxManager, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; +export const checkboxManager = useCheckboxManager(); # Interactive Security Engine Installation Guide @@ -72,8 +73,7 @@ Let's check that CrowdSec is running and able to retrieve the community blocklis #### Enroll your Security Engine into CrowdSec Console For advanced monitoring and trouble shooting the CrowdSec Console is a great tool to visualize your Security Engine's activity and alerts. - - + - [link to doc] - You'll see a confirmation pop up in the console for enrollment if not you might have conectivity issue to the central API [link to troubleshooting section] @@ -83,8 +83,7 @@ For advanced monitoring and trouble shooting the CrowdSec Console is a great too #### CrowdSec installation health - - + ```bash systemctl status crowdsec ``` From 31509e585624a314f6b2107c6e65a41bd6497d1c Mon Sep 17 00:00:00 2001 From: jdv Date: Tue, 10 Jun 2025 17:26:14 +0200 Subject: [PATCH 14/46] reverting properly ? --- .../src/components/InteractiveCheckbox.js | 58 ++++++------------- .../interactive_se_install/01_install.mdx | 9 +-- 2 files changed, 23 insertions(+), 44 deletions(-) diff --git a/crowdsec-docs/src/components/InteractiveCheckbox.js b/crowdsec-docs/src/components/InteractiveCheckbox.js index f11a5e3c..bfdf1b54 100644 --- a/crowdsec-docs/src/components/InteractiveCheckbox.js +++ b/crowdsec-docs/src/components/InteractiveCheckbox.js @@ -3,8 +3,8 @@ import React, { useState, useEffect, createContext, useContext } from 'react'; // Global context to manage checkbox states across all instances const CheckboxContext = createContext(); -// Hook to create a standalone checkbox manager -export const useCheckboxManager = () => { +// Provider component to wrap your MDX content +export const CheckboxProvider = ({ children }) => { const [checkboxStates, setCheckboxStates] = useState({}); const [manualChecks, setManualChecks] = useState({}); @@ -60,21 +60,14 @@ export const useCheckboxManager = () => { })); }; - return { - checkboxStates, - manualChecks, - updateCheckbox, - registerCheckbox, - isCheckboxChecked - }; -}; - -// Provider component to wrap your MDX content (optional) -export const CheckboxProvider = ({ children }) => { - const checkboxManager = useCheckboxManager(); - return ( - + {children} ); @@ -85,17 +78,13 @@ export const InteractiveCheckbox = ({ id, references = [], label = '', - className = '', - manager = null // Accept a manager prop + className = '' }) => { - const contextManager = useContext(CheckboxContext); + const context = useContext(CheckboxContext); - // Use the passed manager or fall back to context, then to a warning fallback - const checkboxManager = manager || contextManager; - - // Fallback if no manager is available - if (!checkboxManager) { - console.warn('InteractiveCheckbox must either be used within a CheckboxProvider or have a manager prop passed to it'); + // Fallback if context is not available (useful for development/SSR) + if (!context) { + console.warn('InteractiveCheckbox must be used within a CheckboxProvider'); return (
{ if (registerCheckbox) { @@ -154,10 +143,7 @@ export const InteractiveCheckbox = ({ }; return ( -
+
{isChecked && ( @@ -192,14 +177,7 @@ export const InteractiveCheckbox = ({ {label && ( diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx index e417dbd2..a824729c 100644 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx +++ b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx @@ -7,8 +7,7 @@ pagination_next: user_guides/interactive_se_install/se_install_02 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; -import { useCheckboxManager, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; -export const checkboxManager = useCheckboxManager(); +import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; # Interactive Security Engine Installation Guide @@ -73,7 +72,8 @@ Let's check that CrowdSec is running and able to retrieve the community blocklis #### Enroll your Security Engine into CrowdSec Console For advanced monitoring and trouble shooting the CrowdSec Console is a great tool to visualize your Security Engine's activity and alerts. - + + - [link to doc] - You'll see a confirmation pop up in the console for enrollment if not you might have conectivity issue to the central API [link to troubleshooting section] @@ -83,7 +83,8 @@ For advanced monitoring and trouble shooting the CrowdSec Console is a great too #### CrowdSec installation health - + + ```bash systemctl status crowdsec ``` From 92c517bd7b5a09de5328dd776dbe75517a90ae75 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 12 Jun 2025 22:36:22 +0200 Subject: [PATCH 15/46] addition of healthcheck --- .../src/components/InteractiveCheckbox.js | 6 +- .../getting_started/healthcheck.md | 208 ++++++++++++++++++ .../getting_started/installation/linux.mdx | 1 + 3 files changed, 211 insertions(+), 4 deletions(-) create mode 100644 crowdsec-docs/unversioned/getting_started/healthcheck.md diff --git a/crowdsec-docs/src/components/InteractiveCheckbox.js b/crowdsec-docs/src/components/InteractiveCheckbox.js index bfdf1b54..ce1cc2ae 100644 --- a/crowdsec-docs/src/components/InteractiveCheckbox.js +++ b/crowdsec-docs/src/components/InteractiveCheckbox.js @@ -143,7 +143,7 @@ export const InteractiveCheckbox = ({ }; return ( -
+
- {label} - + >{label} )}
); diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.md b/crowdsec-docs/unversioned/getting_started/healthcheck.md new file mode 100644 index 00000000..d7a7218e --- /dev/null +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.md @@ -0,0 +1,208 @@ +--- +id: health_check +title: CrowdSec Security Engine Setup Health-Check +--- + +Welcome to the interactive health check of your CrowdSec setup. +We'll have a Top-Down approach: +First, check the proper functioning of the Security Engine's finality: **Detecting** AND **Protecting** against attacks. +Then, eventually, diving into troubleshooting any issue you may encounter. + +## Detection checks + +Check that your Security Engine properly reads and parses the logs of the services you protect. + +### Trigger CrowdSec's test scenarios + +The HTTP collection and the Linux collection contain test scenarios allowing you to prove your Security Engine works as intended, without having to do penetration tests, risking getting you banned from your own server. + +#### Test HTTP detection + +For this test, you'll add the following path to a URL of a website you're protecting: `/CRODWSEC/TEST` + +_*// Any issue possible with .htaccess that would rewrite before this path reaches the log? – **Yes, this is a valid concern. Add a note: "Important: Ensure your web server configuration (e.g., .htaccess) does not rewrite or block this specific URL path `/CRODWSEC/TEST`."***_ + +Looking at your alerts for the specific scenario **crowdsec/test-http**, you should see a line related to your test. + +Note that this scenario has a delay of 1 minute before it can re-trigger _(blackhole parameter of the scenario)_. You can use the "-a" flag to list all alerts + +```bash +sudo cscli alerts list -s crowdsec/test-http -a +``` + +#### Test SSH detection + +Try to authenticate to your server via SSH using the following user: `ABCDEFGH123456` + +Looking at your alerts for the specific scenario **crowdsec/test-ssh**, you should see a line related to your test. + +Note that this scenario has a delay of 5 minute before it can re-trigger _(blackhole parameter of the scenario)_. You can use the "-a" flag to list all alerts + +```bash +sudo cscli alerts list -s crowdsec/test-ssh -a +``` + +#### Test AppSec detection + +If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test it by [insert specific instructions based on typical AppSec setup; see notes below]. + +_[**Important Notes for AppSec Test:** The specific test and instructions here are heavily dependent on the WAF/bouncer being used (nginx-bouncer, cloudflare-bouncer, etc.) and the specific AppSec rules enabled. You need a generic but useful example. Suggestions:* + +* *If possible, install a VERY basic "test XSS" AppSec rule as part of a starter collection.* The test could be adding `` as a query parameter value in your web application. +* *If targeting a specific bouncer/WAF, provide instructions for a known trigger for a common rule.* For example, if using the Cloudflare bouncer, provide a query that triggers a known SQL injection or XSS rule.] + +* * * + +#### Ready for next phase ? + +Were all the tests related to your setup successful ? + +If not, check the troubleshooting section below + +## Detection Troubleshooting
Detection Troubleshooting: No alerts triggered? Let's investigate + +So, you did not see an alert. Here are some tests to identify where the issue might be. + +#### Acquisition and Metrics check: are your logs read and parsed? + +// small description of what acquisition and parser should do + link to docs pages (docs pages to clean up probably soon) + +Let's check the specific metrics section: + +```plain +sudo cscli metrics show acquisition parsers +``` + +If this command fails, check the **CrowdSec Service Troubleshooting section** [
CrowdSec Service Troubleshooting: Command failed? + The `cscli` tool relies on a working CrowdSec service. Possible issues are: + - **CrowdSec not running:** Run `sudo systemctl status crowdsec` to check. Start with `sudo systemctl start crowdsec` if necessary. + - **Permission issues:** Ensure you're running `cscli` commands with `sudo`. + - **Incorrect installation:** If you recently updated or reinstalled CrowdSec, check your installation steps again. + - **LAPI Misconfiguration:** Check /etc/crowdsec/config.yaml for correct LAPI parameters. Check the LAPI connectivity section +
] + +* You should see the names of the log files configured in the acquisition files and the number of lines parsed for each of them. +* The number of _"Lines parsed"_ should be non-zero for each of the files you configured in the acquisition section. +* The parsers metrics show you what parsers were successfully used. Look for the name of the parsers installed for the logs you're reading. + +If you don't see the log names supposed to be parsed, check the **Acquisition Troubleshooting section**[
Acquisition Troubleshooting: Missing Log Files? + Possible issues with missing log files: + - **Incorrect log file path:** Verify the log file path in your `acquis.yaml` file is correct. Typographical errors are common! + - **File permissions:** CrowdSec needs read access to the log files. Ensure the `crowdsec` user (or the user CrowdSec runs as) has permissions. Use `ls -l /path/to/your/log/file` to check. + - **Log file rotation:** If the log file is rotated, CrowdSec may be looking for an older file. Ensure the acquisition configuration is set to follow rotated logs (using wildcards, for example). + - **Log file doesn't exist:** The log file may simply not exist! Verify the service is actually generating logs. + - **Syntax errors:** Validate your acquisition file using `sudo cscli config test`. +
] + +If you don't see parsed lines, check the **Collection Troubleshooting section**[
Collection Troubleshooting: No lines parsed? + Here are a few checks to do to troubleshoot why a given collection of parsers isn't working + - **Log format mismatch:** The log format may not match the parsers in the collection. + - **Install collection**: Verify the hub's command line to install a given collection, there may be missing dependencies or manual actions to trigger +
] + +
+ +## Acquisition Troubleshooting
Acquisition Troubleshooting: Further Steps + +(Flesh out with more detailed steps. Include: + +* Using `cscli explain` to diagnose parsing issues. +* Example `acquis.yaml` configurations for different log sources (files, syslog). +* Guidance on using wildcards and excluding files. +* Common issues: Incorrect log paths, permission errors, file rotation problems.) + +
+ +## Collections Troubleshooting
Collection Troubleshooting: Further Steps + +(Flesh out with more detailed steps. Include: + +* Ensuring the necessary parsers and scenarios are installed. +* Verifying that the log format matches the parsers. +* How to identify relevant collections and dependencies using the hub. +* Addressing errors during collection installation or updates.) + +
+ +## CrowdSec Service Troubleshooting
CrowdSec Service Troubleshooting: Further Steps + +(Expand with more detailed steps. Include: + +* Checking `crowdsec.log` for errors (specific examples of error messages and their meanings). +* Troubleshooting port conflicts (8080, 6060). +* Permission issues for the CrowdSec user. +* Configuration file validation. +* Resource limits (CPU, memory). +* Systemd journal analysis (using `journalctl -u crowdsec`).) + +
+ +## CrowdSec Connectivity checks + +Check that your Security Engine can communicate with the CrowdSec Central API and receive the community blocklist and additional ones. + +// Do we make them check console enroll here? or just cscli capi status? - *Recommend console enrollment here; the previous checks establish core functionality. Enrollment becomes the next logical step.* + +```bash +sudo cscli capi status +``` + +[
What to expect? +- Should show: "You can successfully interact with Central API (CAPI)" +- If this test fail, try to enrol to the console to check what's going on. +
] + +[
Register on the Console +- Link to your documentation +- You could explain how to display all security engines and select the right ones + +
] + +## CrowdSec Network Connectivity Troubleshooting
Network Troubleshooting + +(Expand this section to cover: + +* Firewall rules blocking outbound connections to the CrowdSec Central API. +* DNS resolution issues. +* Proxy server configuration. +* Connectivity issues within Docker containers.) + +
+ +## Remediation checks + +* Canary tester + +(This section requires a robust, safe, and automated canary tester solution. The biggest challenge is preventing users from accidentally banning themselves. This section can be skipped on V0, as you mentioned, and focused on later. For V1, It could be a URL to a third party machine for testing, or a private machine dedicated to running a canary test) + +## Bouncer Troubleshooting
Bouncer Troubleshooting: Steps + +(Expand this section. Include: + +* Verifying that the bouncer is registered and active. +* Checking bouncer logs for errors. +* Troubleshooting communication issues between the bouncer and the CrowdSec LAPI. +* Firewall configuration for the bouncer. +* Specific troubleshooting steps for common bouncers (firewall-bouncer, nginx-bouncer, etc.).) + +
+ +## Profile Troubleshooting
Profile Troubleshooting: Steps + +(Expand this section with more detailed steps. Include: + +* Verifying profile configuration. +* Troubleshooting errors related to profile application. +* Understanding the impact of profiles on detection and remediation.) + +
+ +**Important Considerations:** + +* **Brevity:** Keep the content within the accordion details concise and focused on solving the most common problems. +* **Links to Documentation:** Provide links to the full CrowdSec documentation for more in-depth information. +* **Specificity:** Tailor the troubleshooting steps to the most common issues for each component. +* **Visual Aids:** Use screenshots and code examples to illustrate the troubleshooting steps. +* **Console Verification:** The more checks that can be verified via the console, the better. + +This detailed breakdown should give you a solid foundation for building an effective and user-friendly Health Check for your CrowdSec quick guides. Remember to test the guide thoroughly and gather feedback from users to identify areas for improvement. diff --git a/crowdsec-docs/unversioned/getting_started/installation/linux.mdx b/crowdsec-docs/unversioned/getting_started/installation/linux.mdx index 4ba9c5c8..60161360 100644 --- a/crowdsec-docs/unversioned/getting_started/installation/linux.mdx +++ b/crowdsec-docs/unversioned/getting_started/installation/linux.mdx @@ -8,6 +8,7 @@ pagination_next: getting_started/next_steps import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; +import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; # Installation Linux From 2584b1da75a40bdcf73b5d8d144e25193fb3884a Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 12 Jun 2025 23:18:06 +0200 Subject: [PATCH 16/46] fixed healthcheck v0 --- .../getting_started/healthcheck.md | 208 ------------------ .../getting_started/healthcheck.mdx | 167 ++++++++++++++ 2 files changed, 167 insertions(+), 208 deletions(-) delete mode 100644 crowdsec-docs/unversioned/getting_started/healthcheck.md create mode 100644 crowdsec-docs/unversioned/getting_started/healthcheck.mdx diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.md b/crowdsec-docs/unversioned/getting_started/healthcheck.md deleted file mode 100644 index d7a7218e..00000000 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.md +++ /dev/null @@ -1,208 +0,0 @@ ---- -id: health_check -title: CrowdSec Security Engine Setup Health-Check ---- - -Welcome to the interactive health check of your CrowdSec setup. -We'll have a Top-Down approach: -First, check the proper functioning of the Security Engine's finality: **Detecting** AND **Protecting** against attacks. -Then, eventually, diving into troubleshooting any issue you may encounter. - -## Detection checks - -Check that your Security Engine properly reads and parses the logs of the services you protect. - -### Trigger CrowdSec's test scenarios - -The HTTP collection and the Linux collection contain test scenarios allowing you to prove your Security Engine works as intended, without having to do penetration tests, risking getting you banned from your own server. - -#### Test HTTP detection - -For this test, you'll add the following path to a URL of a website you're protecting: `/CRODWSEC/TEST` - -_*// Any issue possible with .htaccess that would rewrite before this path reaches the log? – **Yes, this is a valid concern. Add a note: "Important: Ensure your web server configuration (e.g., .htaccess) does not rewrite or block this specific URL path `/CRODWSEC/TEST`."***_ - -Looking at your alerts for the specific scenario **crowdsec/test-http**, you should see a line related to your test. - -Note that this scenario has a delay of 1 minute before it can re-trigger _(blackhole parameter of the scenario)_. You can use the "-a" flag to list all alerts - -```bash -sudo cscli alerts list -s crowdsec/test-http -a -``` - -#### Test SSH detection - -Try to authenticate to your server via SSH using the following user: `ABCDEFGH123456` - -Looking at your alerts for the specific scenario **crowdsec/test-ssh**, you should see a line related to your test. - -Note that this scenario has a delay of 5 minute before it can re-trigger _(blackhole parameter of the scenario)_. You can use the "-a" flag to list all alerts - -```bash -sudo cscli alerts list -s crowdsec/test-ssh -a -``` - -#### Test AppSec detection - -If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test it by [insert specific instructions based on typical AppSec setup; see notes below]. - -_[**Important Notes for AppSec Test:** The specific test and instructions here are heavily dependent on the WAF/bouncer being used (nginx-bouncer, cloudflare-bouncer, etc.) and the specific AppSec rules enabled. You need a generic but useful example. Suggestions:* - -* *If possible, install a VERY basic "test XSS" AppSec rule as part of a starter collection.* The test could be adding `` as a query parameter value in your web application. -* *If targeting a specific bouncer/WAF, provide instructions for a known trigger for a common rule.* For example, if using the Cloudflare bouncer, provide a query that triggers a known SQL injection or XSS rule.] - -* * * - -#### Ready for next phase ? - -Were all the tests related to your setup successful ? - -If not, check the troubleshooting section below - -## Detection Troubleshooting
Detection Troubleshooting: No alerts triggered? Let's investigate - -So, you did not see an alert. Here are some tests to identify where the issue might be. - -#### Acquisition and Metrics check: are your logs read and parsed? - -// small description of what acquisition and parser should do + link to docs pages (docs pages to clean up probably soon) - -Let's check the specific metrics section: - -```plain -sudo cscli metrics show acquisition parsers -``` - -If this command fails, check the **CrowdSec Service Troubleshooting section** [
CrowdSec Service Troubleshooting: Command failed? - The `cscli` tool relies on a working CrowdSec service. Possible issues are: - - **CrowdSec not running:** Run `sudo systemctl status crowdsec` to check. Start with `sudo systemctl start crowdsec` if necessary. - - **Permission issues:** Ensure you're running `cscli` commands with `sudo`. - - **Incorrect installation:** If you recently updated or reinstalled CrowdSec, check your installation steps again. - - **LAPI Misconfiguration:** Check /etc/crowdsec/config.yaml for correct LAPI parameters. Check the LAPI connectivity section -
] - -* You should see the names of the log files configured in the acquisition files and the number of lines parsed for each of them. -* The number of _"Lines parsed"_ should be non-zero for each of the files you configured in the acquisition section. -* The parsers metrics show you what parsers were successfully used. Look for the name of the parsers installed for the logs you're reading. - -If you don't see the log names supposed to be parsed, check the **Acquisition Troubleshooting section**[
Acquisition Troubleshooting: Missing Log Files? - Possible issues with missing log files: - - **Incorrect log file path:** Verify the log file path in your `acquis.yaml` file is correct. Typographical errors are common! - - **File permissions:** CrowdSec needs read access to the log files. Ensure the `crowdsec` user (or the user CrowdSec runs as) has permissions. Use `ls -l /path/to/your/log/file` to check. - - **Log file rotation:** If the log file is rotated, CrowdSec may be looking for an older file. Ensure the acquisition configuration is set to follow rotated logs (using wildcards, for example). - - **Log file doesn't exist:** The log file may simply not exist! Verify the service is actually generating logs. - - **Syntax errors:** Validate your acquisition file using `sudo cscli config test`. -
] - -If you don't see parsed lines, check the **Collection Troubleshooting section**[
Collection Troubleshooting: No lines parsed? - Here are a few checks to do to troubleshoot why a given collection of parsers isn't working - - **Log format mismatch:** The log format may not match the parsers in the collection. - - **Install collection**: Verify the hub's command line to install a given collection, there may be missing dependencies or manual actions to trigger -
] - -
- -## Acquisition Troubleshooting
Acquisition Troubleshooting: Further Steps - -(Flesh out with more detailed steps. Include: - -* Using `cscli explain` to diagnose parsing issues. -* Example `acquis.yaml` configurations for different log sources (files, syslog). -* Guidance on using wildcards and excluding files. -* Common issues: Incorrect log paths, permission errors, file rotation problems.) - -
- -## Collections Troubleshooting
Collection Troubleshooting: Further Steps - -(Flesh out with more detailed steps. Include: - -* Ensuring the necessary parsers and scenarios are installed. -* Verifying that the log format matches the parsers. -* How to identify relevant collections and dependencies using the hub. -* Addressing errors during collection installation or updates.) - -
- -## CrowdSec Service Troubleshooting
CrowdSec Service Troubleshooting: Further Steps - -(Expand with more detailed steps. Include: - -* Checking `crowdsec.log` for errors (specific examples of error messages and their meanings). -* Troubleshooting port conflicts (8080, 6060). -* Permission issues for the CrowdSec user. -* Configuration file validation. -* Resource limits (CPU, memory). -* Systemd journal analysis (using `journalctl -u crowdsec`).) - -
- -## CrowdSec Connectivity checks - -Check that your Security Engine can communicate with the CrowdSec Central API and receive the community blocklist and additional ones. - -// Do we make them check console enroll here? or just cscli capi status? - *Recommend console enrollment here; the previous checks establish core functionality. Enrollment becomes the next logical step.* - -```bash -sudo cscli capi status -``` - -[
What to expect? -- Should show: "You can successfully interact with Central API (CAPI)" -- If this test fail, try to enrol to the console to check what's going on. -
] - -[
Register on the Console -- Link to your documentation -- You could explain how to display all security engines and select the right ones - -
] - -## CrowdSec Network Connectivity Troubleshooting
Network Troubleshooting - -(Expand this section to cover: - -* Firewall rules blocking outbound connections to the CrowdSec Central API. -* DNS resolution issues. -* Proxy server configuration. -* Connectivity issues within Docker containers.) - -
- -## Remediation checks - -* Canary tester - -(This section requires a robust, safe, and automated canary tester solution. The biggest challenge is preventing users from accidentally banning themselves. This section can be skipped on V0, as you mentioned, and focused on later. For V1, It could be a URL to a third party machine for testing, or a private machine dedicated to running a canary test) - -## Bouncer Troubleshooting
Bouncer Troubleshooting: Steps - -(Expand this section. Include: - -* Verifying that the bouncer is registered and active. -* Checking bouncer logs for errors. -* Troubleshooting communication issues between the bouncer and the CrowdSec LAPI. -* Firewall configuration for the bouncer. -* Specific troubleshooting steps for common bouncers (firewall-bouncer, nginx-bouncer, etc.).) - -
- -## Profile Troubleshooting
Profile Troubleshooting: Steps - -(Expand this section with more detailed steps. Include: - -* Verifying profile configuration. -* Troubleshooting errors related to profile application. -* Understanding the impact of profiles on detection and remediation.) - -
- -**Important Considerations:** - -* **Brevity:** Keep the content within the accordion details concise and focused on solving the most common problems. -* **Links to Documentation:** Provide links to the full CrowdSec documentation for more in-depth information. -* **Specificity:** Tailor the troubleshooting steps to the most common issues for each component. -* **Visual Aids:** Use screenshots and code examples to illustrate the troubleshooting steps. -* **Console Verification:** The more checks that can be verified via the console, the better. - -This detailed breakdown should give you a solid foundation for building an effective and user-friendly Health Check for your CrowdSec quick guides. Remember to test the guide thoroughly and gather feedback from users to identify areas for improvement. diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx new file mode 100644 index 00000000..6ccb6eaa --- /dev/null +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx @@ -0,0 +1,167 @@ +--- +id: health_check +title: CrowdSec Security Engine Setup Health-Check +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; +import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; + +Welcome to the interactive health check of your CrowdSec setup. +We'll have a Top-Down approach: +First, check the proper functioning of the Security Engine's finality: **Detecting** AND **Protecting** against attacks. +Then, eventually, diving into troubleshooting any issue you may encounter. + +## Detection checks + +Check that your Security Engine properly reads and parses the logs of the services you protect. + +### Trigger CrowdSec's test scenarios + +The HTTP collection and the Linux collection contain test scenarios allowing you to prove your Security Engine works as intended, without having to do penetration tests, risking getting you banned from your own server. + +#### Test HTTP detection + +For this test, you'll add the following path to a URL of a website you're protecting: `/CRODWSEC/TEST` +_*// Any issue possible with .htaccess that would rewrite before this path reaches the log? – **Yes, this is a valid concern. Add a note: "Important: Ensure your web server configuration (e.g., .htaccess) does not rewrite or block this specific URL path `/CRODWSEC/TEST`."***_ +Looking at your alerts for the specific scenario **crowdsec/test-http**, you should see a line related to your test. +Note that this scenario has a delay of 1 minute before it can re-trigger _(blackhole parameter of the scenario)_. You can use the "-a" flag to list all alerts. + +sudo cscli alerts list -s crowdsec/test-http -a + +#### Test SSH detection + +Try to authenticate to your server via SSH using the following user: `ABCDEFGH123456`. +Looking at your alerts for the specific scenario **crowdsec/test-ssh**, you should see a line related to your test. +Note that this scenario has a delay of 5 minute before it can re-trigger _(blackhole parameter of the scenario)_. You can use the "-a" flag to list all alerts. + +sudo cscli alerts list -s crowdsec/test-ssh -a + +#### Test AppSec detection + +If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test it by [insert specific instructions based on typical AppSec setup; see notes below]. +_[**Important Notes for AppSec Test:** The specific test and instructions here are heavily dependent on the WAF/bouncer being used (nginx-bouncer, cloudflare-bouncer, etc.) and the specific AppSec rules enabled. You need a generic but useful example. Suggestions:* +- *If possible, install a VERY basic "test XSS" AppSec rule as part of a starter collection.* The test could be adding `` as a query parameter value in your web application. +- *If targeting a specific bouncer/WAF, provide instructions for a known trigger for a common rule.* For example, if using the Cloudflare bouncer, provide a query that triggers a known SQL injection or XSS rule.] + +* * * + +#### Ready for next phase ? + +Were all the tests related to your setup successful ? +If not, check the troubleshooting section below + +### Detection Troubleshooting + +No alerts triggered? Let's investigate: Here are some tests to identify where the issue might be. + +
+ Acquisition and Metrics check: are your logs read and parsed? + +// small description of what acquisition and parser should do + link to docs pages (docs pages to clean up probably soon) +Let's check the specific metrics section: + +sudo cscli metrics show acquisition parsers + +- You should see the names of the log files/stream configured in the acquisition files, and the number of lines parsed for each of them. +- The number of "Lines parsed" should be non-zero for each of the files you configured in the acquisition section. +- The parsers metrics show you what parsers were successfully used. Look for the name of the parsers installed for the logs you're reading + +
+ 🚨 If this command fails, check the [**CrowdSec Service Troubleshooting section**](#crowdsec_service_troubleshooting) +
+
+ 🚨 If you don't see the log names supposed to be parsed, check the [**Acquisition Troubleshooting section**](#acquisition_troubleshooting) +
+
+ 🚨 If you don't see parsed lines, check the [**Collection Troubleshooting section****](#collection_troubleshooting) +
+ +
+ +
+ Acquisition Troubleshooting: are your logs properly declared as datasources +Your acquisition configuration is located in `/etc/crowdsec/acquis.yaml` or in `/etc/crowdsec/acquis.d/anyName.yaml`. +- Check that the datasource is properly setup: file path or correct port for syslog server... +- Ensure that the type declared matches the parser expected to be used: nginx, apache, syslog, etc. +- for more information, check the [documentation for the various datasources](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro) +
+ +
+ Collection Troubleshooting: are the parsers and scenarios installed? +#### Collection installation +- On host installations your existing service should have been detected and appropriate collections (containing parsers and scenarios) installed automatically. +- How ever on other kind of installations, or in case of failed detection, you may need to install the collections manually. + +Check the installed collections with the command: +sudo cscli collections list +You can also list individual parsers and scenarios with: +sudo cscli parsers list +sudo cscli scenarios list +- If you see the parsers and scenarios related to your service, then you're good to go. + +- To install new collection, check the [CrowdSec Hub](https://hub.crowdsec.net/) for a collection named after your service (e.g., [nginx](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx), [apache](https://app.crowdsec.net/hub/author/crowdsecurity/collections/apache), [linux](https://app.crowdsec.net/hub/author/crowdsecurity/collections/linux), etc.). +- If you find the collection, install it using the command shown on the hub page and follow the instructions provided for the acquisition file. + +#### Log format mismatch +- If your logs are in a custom format, you may need to create your own parsers or modify the existing ones to match your log format. +- You can find more information on how to create your own parsers in the [CrowdSec documentation](https://doc.crowdsec.net/docs/next/log_processor/parsers/format). + +
+ +
+ CrowdSec Service Troubleshooting: is the CrowdSec service running? + + + +
+ +## CrowdSec Connectivity checks + +Check that your Security Engine can communicate with the CrowdSec Central API and receive the community blocklist and additional ones. + +// Do we make them check console enroll here? or just cscli capi status? - *Recommend console enrollment here; the previous checks establish core functionality. Enrollment becomes the next logical step.* + +```bash +sudo cscli capi status +``` +- Should show: "You can successfully interact with Central API (CAPI)" +- If this test fail, try to enrol to the console to check what's going on. + +
+ Network Connectivity Troubleshooting + +* online api credentials exist in `/etc/crowdsec/config/online_api_credentials.yaml` (default path) +* Firewall rules blocking outbound connections to the CrowdSec Central API. +* DNS resolution issues. +* Proxy server configuration. +* Connectivity issues within Docker containers. + +
+ +## Remediation checks + +* Canary tester + +(This section requires a robust, safe, and automated canary tester solution. The biggest challenge is preventing users from accidentally banning themselves. This section can be skipped on V0, as you mentioned, and focused on later. For V1, It could be a URL to a third party machine for testing, or a private machine dedicated to running a canary test) + +
+ Bouncer Troubleshooting + +* Verifying that the bouncer is registered and active. +* Checking bouncer logs for errors. +* Troubleshooting communication issues between the bouncer and the CrowdSec LAPI. +* Firewall configuration for the bouncer. +* Specific troubleshooting steps for common bouncers (firewall-bouncer, nginx-bouncer, etc.).) + +
+ +
+ Profile Troubleshooting: Steps + +* Verifying profile configuration. +* Troubleshooting errors related to profile application. +* Understanding the impact of profiles on detection and remediation.) + +
\ No newline at end of file From 2e0c8f1ccf6969a900e75c27710761c3084344c9 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 12 Jun 2025 23:27:58 +0200 Subject: [PATCH 17/46] trying to fix style thing ? --- crowdsec-docs/unversioned/getting_started/healthcheck.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx index 6ccb6eaa..82e73e8f 100644 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx @@ -68,13 +68,13 @@ Let's check the specific metrics section: - The number of "Lines parsed" should be non-zero for each of the files you configured in the acquisition section. - The parsers metrics show you what parsers were successfully used. Look for the name of the parsers installed for the logs you're reading -
+
🚨 If this command fails, check the [**CrowdSec Service Troubleshooting section**](#crowdsec_service_troubleshooting)
-
+
🚨 If you don't see the log names supposed to be parsed, check the [**Acquisition Troubleshooting section**](#acquisition_troubleshooting)
-
+
🚨 If you don't see parsed lines, check the [**Collection Troubleshooting section****](#collection_troubleshooting)
From a8a305558db2653b04f6cb499362294a58316611 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 12 Jun 2025 23:36:20 +0200 Subject: [PATCH 18/46] fixed soon to die js module --- crowdsec-docs/src/components/InteractiveCheckbox.js | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/crowdsec-docs/src/components/InteractiveCheckbox.js b/crowdsec-docs/src/components/InteractiveCheckbox.js index ce1cc2ae..2878092c 100644 --- a/crowdsec-docs/src/components/InteractiveCheckbox.js +++ b/crowdsec-docs/src/components/InteractiveCheckbox.js @@ -143,7 +143,7 @@ export const InteractiveCheckbox = ({ }; return ( -
+
{label} + >{label} + )}
); From 2caed79fa609222718555aa0e79b220b6eae2678 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 12 Jun 2025 23:53:48 +0200 Subject: [PATCH 19/46] some mini checks --- .../getting_started/healthcheck.mdx | 27 +++++++++++-------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx index 82e73e8f..b38c9673 100644 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx @@ -13,15 +13,14 @@ We'll have a Top-Down approach: First, check the proper functioning of the Security Engine's finality: **Detecting** AND **Protecting** against attacks. Then, eventually, diving into troubleshooting any issue you may encounter. -## Detection checks - -Check that your Security Engine properly reads and parses the logs of the services you protect. - -### Trigger CrowdSec's test scenarios +## 📡 Detection checks: Trigger CrowdSec's test scenarios +:::info +Check that your Security Engine properly reads and parses the logs of the services you protect. The HTTP collection and the Linux collection contain test scenarios allowing you to prove your Security Engine works as intended, without having to do penetration tests, risking getting you banned from your own server. +::: -#### Test HTTP detection +### Test HTTP detection For this test, you'll add the following path to a URL of a website you're protecting: `/CRODWSEC/TEST` _*// Any issue possible with .htaccess that would rewrite before this path reaches the log? – **Yes, this is a valid concern. Add a note: "Important: Ensure your web server configuration (e.g., .htaccess) does not rewrite or block this specific URL path `/CRODWSEC/TEST`."***_ @@ -30,7 +29,7 @@ Note that this scenario has a delay of 1 minute before it can re-trigger _(black sudo cscli alerts list -s crowdsec/test-http -a -#### Test SSH detection +### Test SSH detection Try to authenticate to your server via SSH using the following user: `ABCDEFGH123456`. Looking at your alerts for the specific scenario **crowdsec/test-ssh**, you should see a line related to your test. @@ -38,7 +37,7 @@ Note that this scenario has a delay of 5 minute before it can re-trigger _(black sudo cscli alerts list -s crowdsec/test-ssh -a -#### Test AppSec detection +### Test AppSec detection If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test it by [insert specific instructions based on typical AppSec setup; see notes below]. _[**Important Notes for AppSec Test:** The specific test and instructions here are heavily dependent on the WAF/bouncer being used (nginx-bouncer, cloudflare-bouncer, etc.) and the specific AppSec rules enabled. You need a generic but useful example. Suggestions:* @@ -47,9 +46,11 @@ _[**Important Notes for AppSec Test:** The specific test and instructions here a * * * -#### Ready for next phase ? +### Ready for next phase ? + +Were all the tests related to your setup successful ? +If so, you can proceed to the next phase of the health check: [**Connectivity checks**](#crowdsec-connectivity-checks). -Were all the tests related to your setup successful ? If not, check the troubleshooting section below ### Detection Troubleshooting @@ -59,7 +60,8 @@ No alerts triggered? Let's investigate: Here are some tests to identify where th
Acquisition and Metrics check: are your logs read and parsed? -// small description of what acquisition and parser should do + link to docs pages (docs pages to clean up probably soon) +blabla...small description of what acquisition and parser should do + link to docs pages (docs pages to clean up probably soon) + Let's check the specific metrics section: sudo cscli metrics show acquisition parsers @@ -82,14 +84,17 @@ Let's check the specific metrics section:
Acquisition Troubleshooting: are your logs properly declared as datasources + Your acquisition configuration is located in `/etc/crowdsec/acquis.yaml` or in `/etc/crowdsec/acquis.d/anyName.yaml`. - Check that the datasource is properly setup: file path or correct port for syslog server... - Ensure that the type declared matches the parser expected to be used: nginx, apache, syslog, etc. - for more information, check the [documentation for the various datasources](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro) +
Collection Troubleshooting: are the parsers and scenarios installed? + #### Collection installation - On host installations your existing service should have been detected and appropriate collections (containing parsers and scenarios) installed automatically. - How ever on other kind of installations, or in case of failed detection, you may need to install the collections manually. From 3a01af976f5863f03df82480494a1fa4327bacd2 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 12 Jun 2025 23:55:42 +0200 Subject: [PATCH 20/46] nano changes --- crowdsec-docs/unversioned/getting_started/healthcheck.mdx | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx index b38c9673..460487cf 100644 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx @@ -46,16 +46,15 @@ _[**Important Notes for AppSec Test:** The specific test and instructions here a * * * -### Ready for next phase ? +## Ready for next phase ? Were all the tests related to your setup successful ? If so, you can proceed to the next phase of the health check: [**Connectivity checks**](#crowdsec-connectivity-checks). If not, check the troubleshooting section below -### Detection Troubleshooting - -No alerts triggered? Let's investigate: Here are some tests to identify where the issue might be. +## 🚨 Detection Troubleshooting +*No alerts triggered? Let's investigate: Here are some tests to identify where the issue might be.*
Acquisition and Metrics check: are your logs read and parsed? From 8de15a331f6dc88af187356d28487cbf7dbdf245 Mon Sep 17 00:00:00 2001 From: jdv Date: Fri, 13 Jun 2025 00:08:53 +0200 Subject: [PATCH 21/46] testing hierarchical hr list --- .../src/components/HierarchicalHrList.js | 73 +++++++++++++++++++ .../getting_started/healthcheck.mdx | 36 ++++++++- 2 files changed, 107 insertions(+), 2 deletions(-) create mode 100644 crowdsec-docs/src/components/HierarchicalHrList.js diff --git a/crowdsec-docs/src/components/HierarchicalHrList.js b/crowdsec-docs/src/components/HierarchicalHrList.js new file mode 100644 index 00000000..43c3a10e --- /dev/null +++ b/crowdsec-docs/src/components/HierarchicalHrList.js @@ -0,0 +1,73 @@ +import React, { useState } from 'react'; + +const HierarchicalList = ({ data = [] }) => { + const [activeItem, setActiveItem] = useState(null); + + const handleItemClick = (itemId) => { + setActiveItem(activeItem === itemId ? null : itemId); + }; + + return ( +
+

Hierarchical List Example

+ + {/* Main list */} +
+ {data.map((item, index) => ( + +
handleItemClick(item.id)} + > +
+ {item.label} +
+ {index < data.length - 1 && ( +
—
+ )} + + ))} +
+ + {/* Sub-elements */} + {activeItem && ( +
+
+ {data + .find(item => item.id === activeItem) + ?.children?.map((child, index, array) => ( + +
+
+ + {child.label} + +
+ {index < array.length - 1 && ( +
—
+ )} + + ))} +
+
+ )} + + {/* Instructions */} +
+

How to use:

+

+ Click on any main list item to toggle its sub-elements. The sub-elements will appear below + the main list, scaled down by 50% and displayed inline with connecting dashes. +

+
+
+ ); +}; + +export default HierarchicalList; \ No newline at end of file diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx index 460487cf..7afba54c 100644 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx @@ -7,12 +7,44 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; +import HierarchicalList from '@site/src/components/HierarchicalList.js'; -Welcome to the interactive health check of your CrowdSec setup. -We'll have a Top-Down approach: +Welcome to the interactive health check of your CrowdSec setup. We'll have a Top-Down approach: First, check the proper functioning of the Security Engine's finality: **Detecting** AND **Protecting** against attacks. Then, eventually, diving into troubleshooting any issue you may encounter. +; + ## 📡 Detection checks: Trigger CrowdSec's test scenarios :::info From 71423d861dc466626b33646a561137692894bbf2 Mon Sep 17 00:00:00 2001 From: jdv Date: Fri, 13 Jun 2025 00:17:35 +0200 Subject: [PATCH 22/46] nano fix --- .../getting_started/healthcheck.mdx | 64 ++++++++++--------- 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx index 7afba54c..6fbb1570 100644 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx @@ -13,39 +13,41 @@ Welcome to the interactive health check of your CrowdSec setup. We'll have a Top First, check the proper functioning of the Security Engine's finality: **Detecting** AND **Protecting** against attacks. Then, eventually, diving into troubleshooting any issue you may encounter. -; + }, + { + id: 2, + label: 'Connectivity', + color: 'bg-green-500', + children: [ + { id: 21, label: 'Connectivity checks', color: 'bg-green-300' }, + { id: 22, label: 'Troubleshooting', color: 'bg-green-300' } + ] + }, + { + id: 3, + label: 'Remediation', + color: 'bg-purple-500', + children: [ + { id: 31, label: 'Remediation checks', color: 'bg-purple-300' }, + { id: 32, label: 'Troubleshooting', color: 'bg-purple-300' } + ] + } +]; + + + +## 📡 Detection checks -## 📡 Detection checks: Trigger CrowdSec's test scenarios +*Trigger CrowdSec's test scenarios* :::info Check that your Security Engine properly reads and parses the logs of the services you protect. From ecda6972647bab04dd49efc3e2594132442ad232 Mon Sep 17 00:00:00 2001 From: jdv Date: Fri, 13 Jun 2025 00:19:53 +0200 Subject: [PATCH 23/46] nano fix --- .../src/components/HierarchicalHrList.js | 73 ------------------- 1 file changed, 73 deletions(-) delete mode 100644 crowdsec-docs/src/components/HierarchicalHrList.js diff --git a/crowdsec-docs/src/components/HierarchicalHrList.js b/crowdsec-docs/src/components/HierarchicalHrList.js deleted file mode 100644 index 43c3a10e..00000000 --- a/crowdsec-docs/src/components/HierarchicalHrList.js +++ /dev/null @@ -1,73 +0,0 @@ -import React, { useState } from 'react'; - -const HierarchicalList = ({ data = [] }) => { - const [activeItem, setActiveItem] = useState(null); - - const handleItemClick = (itemId) => { - setActiveItem(activeItem === itemId ? null : itemId); - }; - - return ( -
-

Hierarchical List Example

- - {/* Main list */} -
- {data.map((item, index) => ( - -
handleItemClick(item.id)} - > -
- {item.label} -
- {index < data.length - 1 && ( -
—
- )} - - ))} -
- - {/* Sub-elements */} - {activeItem && ( -
-
- {data - .find(item => item.id === activeItem) - ?.children?.map((child, index, array) => ( - -
-
- - {child.label} - -
- {index < array.length - 1 && ( -
—
- )} - - ))} -
-
- )} - - {/* Instructions */} -
-

How to use:

-

- Click on any main list item to toggle its sub-elements. The sub-elements will appear below - the main list, scaled down by 50% and displayed inline with connecting dashes. -

-
-
- ); -}; - -export default HierarchicalList; \ No newline at end of file From 1ad4145da29e686da783acfd244eee2ded565685 Mon Sep 17 00:00:00 2001 From: jdv Date: Fri, 13 Jun 2025 00:22:12 +0200 Subject: [PATCH 24/46] nano fix --- .../src/components/HierarchicalList.js | 73 +++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 crowdsec-docs/src/components/HierarchicalList.js diff --git a/crowdsec-docs/src/components/HierarchicalList.js b/crowdsec-docs/src/components/HierarchicalList.js new file mode 100644 index 00000000..43c3a10e --- /dev/null +++ b/crowdsec-docs/src/components/HierarchicalList.js @@ -0,0 +1,73 @@ +import React, { useState } from 'react'; + +const HierarchicalList = ({ data = [] }) => { + const [activeItem, setActiveItem] = useState(null); + + const handleItemClick = (itemId) => { + setActiveItem(activeItem === itemId ? null : itemId); + }; + + return ( +
+

Hierarchical List Example

+ + {/* Main list */} +
+ {data.map((item, index) => ( + +
handleItemClick(item.id)} + > +
+ {item.label} +
+ {index < data.length - 1 && ( +
—
+ )} + + ))} +
+ + {/* Sub-elements */} + {activeItem && ( +
+
+ {data + .find(item => item.id === activeItem) + ?.children?.map((child, index, array) => ( + +
+
+ + {child.label} + +
+ {index < array.length - 1 && ( +
—
+ )} + + ))} +
+
+ )} + + {/* Instructions */} +
+

How to use:

+

+ Click on any main list item to toggle its sub-elements. The sub-elements will appear below + the main list, scaled down by 50% and displayed inline with connecting dashes. +

+
+
+ ); +}; + +export default HierarchicalList; \ No newline at end of file From be40c54d9f47cb8020e0f215dc115688877e3ec2 Mon Sep 17 00:00:00 2001 From: jdv Date: Fri, 13 Jun 2025 00:32:01 +0200 Subject: [PATCH 25/46] no hierarchical list anymore, it's bugged --- .../src/components/HierarchicalList.js | 4 +-- .../getting_started/healthcheck.mdx | 32 ------------------- 2 files changed, 1 insertion(+), 35 deletions(-) diff --git a/crowdsec-docs/src/components/HierarchicalList.js b/crowdsec-docs/src/components/HierarchicalList.js index 43c3a10e..fba6250a 100644 --- a/crowdsec-docs/src/components/HierarchicalList.js +++ b/crowdsec-docs/src/components/HierarchicalList.js @@ -8,9 +8,7 @@ const HierarchicalList = ({ data = [] }) => { }; return ( -
-

Hierarchical List Example

- +
{/* Main list */}
{data.map((item, index) => ( diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx index 6fbb1570..dd9097ab 100644 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx @@ -13,38 +13,6 @@ Welcome to the interactive health check of your CrowdSec setup. We'll have a Top First, check the proper functioning of the Security Engine's finality: **Detecting** AND **Protecting** against attacks. Then, eventually, diving into troubleshooting any issue you may encounter. -export const diagnosticData = [ - { - id: 1, - label: 'Detection', - color: 'bg-blue-500', - children: [ - { id: 11, label: 'Detection checks', color: 'bg-blue-300' }, - { id: 12, label: 'Troubleshooting', color: 'bg-blue-300' } - ] - }, - { - id: 2, - label: 'Connectivity', - color: 'bg-green-500', - children: [ - { id: 21, label: 'Connectivity checks', color: 'bg-green-300' }, - { id: 22, label: 'Troubleshooting', color: 'bg-green-300' } - ] - }, - { - id: 3, - label: 'Remediation', - color: 'bg-purple-500', - children: [ - { id: 31, label: 'Remediation checks', color: 'bg-purple-300' }, - { id: 32, label: 'Troubleshooting', color: 'bg-purple-300' } - ] - } -]; - - - ## 📡 Detection checks *Trigger CrowdSec's test scenarios* From ef6598bbf764dc8adc43e8284c79d6ae0130ee68 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 09:59:11 +0200 Subject: [PATCH 26/46] detailsin details test - basic for now --- .../getting_started/healthcheck.mdx | 167 +++++++++++------- 1 file changed, 100 insertions(+), 67 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx index dd9097ab..19b89026 100644 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx @@ -10,117 +10,150 @@ import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/Inte import HierarchicalList from '@site/src/components/HierarchicalList.js'; Welcome to the interactive health check of your CrowdSec setup. We'll have a Top-Down approach: -First, check the proper functioning of the Security Engine's finality: **Detecting** AND **Protecting** against attacks. -Then, eventually, diving into troubleshooting any issue you may encounter. +First, check the proper functioning of the Security Engine's finality: +- **📡 Detecting** behaviors on your services. +- **🔗 Connectivity** with CrowdSec network to retrieve threats lists. +- **🛡️ Protecting** your services by remediating alerts automatically with bouncers. + +If the initial checks are not conclusive, we'll dive into a comprehensive troubleshooting for each section. ## 📡 Detection checks -*Trigger CrowdSec's test scenarios* +### *Trigger CrowdSec's test scenarios* :::info Check that your Security Engine properly reads and parses the logs of the services you protect. -The HTTP collection and the Linux collection contain test scenarios allowing you to prove your Security Engine works as intended, without having to do penetration tests, risking getting you banned from your own server. +The HTTP collection and the Linux collection contain **dummy scenarios** allowing you to prove your Security Engine works as intended, without having to do penetration tests, risking getting you banned from your own server. ::: -### Test HTTP detection +
+ ☑️ Test HTTP detection +Let's trigger the `crowdsecurity/http-generic-test` dummy scenario by calling a *probe path* on your web server. + +1️⃣ Request your service URL with the following path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` +curl -I http://your-service-url/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl + +2️⃣ You should see an alert for the scenario `crowdsecurity/http-generic-test` + +sudo cscli alerts list -s crowdsecurity/http-generic-test + +Note that this scenario has a delay of 5 minutes before it can re-trigger _(blackhole parameter of the scenario)_. +
+ +
+ ☑️ Test SSH detection +Let's trigger the `crowdsecurity/ssh-generic-test` dummy scenario by logging in to your server via SSH with a specific user. + +1️⃣ Try to authenticate to your server via SSH using the following user: `crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl`. +ssh crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl@\ + +2️⃣ You should see an alert for the scenario `crowdsecurity/ssh-generic-test` + +sudo cscli alerts list -s crowdsecurity/ssh-generic-test -For this test, you'll add the following path to a URL of a website you're protecting: `/CRODWSEC/TEST` -_*// Any issue possible with .htaccess that would rewrite before this path reaches the log? – **Yes, this is a valid concern. Add a note: "Important: Ensure your web server configuration (e.g., .htaccess) does not rewrite or block this specific URL path `/CRODWSEC/TEST`."***_ -Looking at your alerts for the specific scenario **crowdsec/test-http**, you should see a line related to your test. -Note that this scenario has a delay of 1 minute before it can re-trigger _(blackhole parameter of the scenario)_. You can use the "-a" flag to list all alerts. +Note that this scenario has a delay of 5 minute before it can re-trigger _(blackhole parameter of the scenario)_. +
-sudo cscli alerts list -s crowdsec/test-http -a +
+ ☑️ Test CrowdSec WAF with an AppSec detection +If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test the detection of an AppSec scenario. +It's similar to the HTTP detection test, but it will trigger `crowdsecurity/appsec-generic-test`. -### Test SSH detection +Here is how to trigger the `crowdsecurity/appsec-generic-test` dummy scenario by calling a *probe path* on your web server. -Try to authenticate to your server via SSH using the following user: `ABCDEFGH123456`. -Looking at your alerts for the specific scenario **crowdsec/test-ssh**, you should see a line related to your test. -Note that this scenario has a delay of 5 minute before it can re-trigger _(blackhole parameter of the scenario)_. You can use the "-a" flag to list all alerts. +1️⃣ Request your service URL with the following path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` +curl -I http://your-service-url/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl -sudo cscli alerts list -s crowdsec/test-ssh -a +2️⃣ You should see an alert for the scenario `crowdsecurity/appsec-generic-test` -### Test AppSec detection +sudo cscli alerts list -s crowdsecurity/appsec-generic-test -If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test it by [insert specific instructions based on typical AppSec setup; see notes below]. -_[**Important Notes for AppSec Test:** The specific test and instructions here are heavily dependent on the WAF/bouncer being used (nginx-bouncer, cloudflare-bouncer, etc.) and the specific AppSec rules enabled. You need a generic but useful example. Suggestions:* -- *If possible, install a VERY basic "test XSS" AppSec rule as part of a starter collection.* The test could be adding `` as a query parameter value in your web application. -- *If targeting a specific bouncer/WAF, provide instructions for a known trigger for a common rule.* For example, if using the Cloudflare bouncer, provide a query that triggers a known SQL injection or XSS rule.] +Note that this scenario has a delay of **1** minute before it can re-trigger _(blackhole parameter of the scenario)_. +
* * * -## Ready for next phase ? +### Were all the tests successful ? Were all the tests related to your setup successful ? If so, you can proceed to the next phase of the health check: [**Connectivity checks**](#crowdsec-connectivity-checks). If not, check the troubleshooting section below -## 🚨 Detection Troubleshooting -*No alerts triggered? Let's investigate: Here are some tests to identify where the issue might be.* -
- Acquisition and Metrics check: are your logs read and parsed? + 🚨 Detection Troubleshooting -blabla...small description of what acquisition and parser should do + link to docs pages (docs pages to clean up probably soon) + *No alerts triggered? Let's investigate: Here are some tests to identify where the issue might be.* + If you installed CrowdSec Security Engine on the same host as the service you want to protect, the install wizards should have automatically detected the service and installed the appropriate parsers and scenarios. + If you have non-default paths or format for your logs, or if you chose other installation methods (docker, kubernetes..), you may need to manually install the parsers and scenarios. + + **This troubleshooting section will help you identify the issue and guide you through the necessary steps to fix it.** -Let's check the specific metrics section: +
+ Acquisition metrics check: are your logs read and parsed? -sudo cscli metrics show acquisition parsers + The acquisition and parsing aspect of CrowdSec is crucial, as it tells The Security Engine which logs to read and how to parse them. + You can setup multiple datasources (files, syslog, etc.), for more details you can refer to the [datasources documentation](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro). -- You should see the names of the log files/stream configured in the acquisition files, and the number of lines parsed for each of them. -- The number of "Lines parsed" should be non-zero for each of the files you configured in the acquisition section. -- The parsers metrics show you what parsers were successfully used. Look for the name of the parsers installed for the logs you're reading + Let's do a Top Down check using the `cscli metrics` command to see if your logs are being read and parsed correctly. -
- 🚨 If this command fails, check the [**CrowdSec Service Troubleshooting section**](#crowdsec_service_troubleshooting) -
-
- 🚨 If you don't see the log names supposed to be parsed, check the [**Acquisition Troubleshooting section**](#acquisition_troubleshooting) -
-
- 🚨 If you don't see parsed lines, check the [**Collection Troubleshooting section****](#collection_troubleshooting) -
+ sudo cscli metrics show acquisition parsers -
+ - You should see the names of the log files/stream configured in the acquisition files, and the number of lines parsed for each of them. + - The number of "Lines parsed" should be non-zero for each of the files you configured in the acquisition section. + - The parsers metrics show you what parsers were successfully used. Look for the name of the parsers installed for the logs you're reading -
- Acquisition Troubleshooting: are your logs properly declared as datasources +
+ 🚨 If this command fails, check the [**CrowdSec Service Troubleshooting section**](#crowdsec_service_troubleshooting) +
+
+ 🚨 If you don't see the log names supposed to be parsed, check the [**Acquisition Troubleshooting section**](#acquisition_troubleshooting) +
+
+ 🚨 If you don't see parsed lines, check the [**Collection Troubleshooting section****](#collection_troubleshooting) +
-Your acquisition configuration is located in `/etc/crowdsec/acquis.yaml` or in `/etc/crowdsec/acquis.d/anyName.yaml`. -- Check that the datasource is properly setup: file path or correct port for syslog server... -- Ensure that the type declared matches the parser expected to be used: nginx, apache, syslog, etc. -- for more information, check the [documentation for the various datasources](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro) +
-
+
+ Acquisition Troubleshooting: are your logs properly declared as datasources -
- Collection Troubleshooting: are the parsers and scenarios installed? + Your acquisition configuration is located in `/etc/crowdsec/acquis.yaml` or in `/etc/crowdsec/acquis.d/anyName.yaml`. + - Check that the datasource is properly setup: file path or correct port for syslog server... + - Ensure that the type declared matches the parser expected to be used: nginx, apache, syslog, etc. + - for more information, check the [documentation for the various datasources](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro) -#### Collection installation -- On host installations your existing service should have been detected and appropriate collections (containing parsers and scenarios) installed automatically. -- How ever on other kind of installations, or in case of failed detection, you may need to install the collections manually. +
-Check the installed collections with the command: -sudo cscli collections list -You can also list individual parsers and scenarios with: -sudo cscli parsers list -sudo cscli scenarios list -- If you see the parsers and scenarios related to your service, then you're good to go. +
+ Collection Troubleshooting: are the parsers and scenarios installed? -- To install new collection, check the [CrowdSec Hub](https://hub.crowdsec.net/) for a collection named after your service (e.g., [nginx](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx), [apache](https://app.crowdsec.net/hub/author/crowdsecurity/collections/apache), [linux](https://app.crowdsec.net/hub/author/crowdsecurity/collections/linux), etc.). -- If you find the collection, install it using the command shown on the hub page and follow the instructions provided for the acquisition file. + #### Collection installation + - On host installations your existing service should have been detected and appropriate collections (containing parsers and scenarios) installed automatically. + - How ever on other kind of installations, or in case of failed detection, you may need to install the collections manually. -#### Log format mismatch -- If your logs are in a custom format, you may need to create your own parsers or modify the existing ones to match your log format. -- You can find more information on how to create your own parsers in the [CrowdSec documentation](https://doc.crowdsec.net/docs/next/log_processor/parsers/format). + Check the installed collections with the command: + sudo cscli collections list + You can also list individual parsers and scenarios with: + sudo cscli parsers list + sudo cscli scenarios list + - If you see the parsers and scenarios related to your service, then you're good to go. -
+ - To install new collection, check the [CrowdSec Hub](https://hub.crowdsec.net/) for a collection named after your service (e.g., [nginx](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx), [apache](https://app.crowdsec.net/hub/author/crowdsecurity/collections/apache), [linux](https://app.crowdsec.net/hub/author/crowdsecurity/collections/linux), etc.). + - If you find the collection, install it using the command shown on the hub page and follow the instructions provided for the acquisition file. -
- CrowdSec Service Troubleshooting: is the CrowdSec service running? + #### Log format mismatch + - If your logs are in a custom format, you may need to create your own parsers or modify the existing ones to match your log format. + - You can find more information on how to create your own parsers in the [CrowdSec documentation](https://doc.crowdsec.net/docs/next/log_processor/parsers/format). + +
+ +
+ CrowdSec Service Troubleshooting: is the CrowdSec service running? +
## CrowdSec Connectivity checks From d4fe6be98e22f0095cbf3c7a46ee0f3969e00c67 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 10:17:57 +0200 Subject: [PATCH 27/46] testing summary without styling --- crowdsec-docs/unversioned/getting_started/healthcheck.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx index 19b89026..2dd0c37a 100644 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx @@ -27,11 +27,11 @@ The HTTP collection and the Linux collection contain **dummy scenarios** allowin :::
- ☑️ Test HTTP detection + Test **HTTP** detection Let's trigger the `crowdsecurity/http-generic-test` dummy scenario by calling a *probe path* on your web server. 1️⃣ Request your service URL with the following path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` -curl -I http://your-service-url/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl +curl -I https://\/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl 2️⃣ You should see an alert for the scenario `crowdsecurity/http-generic-test` @@ -41,7 +41,7 @@ Note that this scenario has a delay of 5 minutes before it can re-trigger _(blac
- ☑️ Test SSH detection + ☑️ Test **SSH** detection Let's trigger the `crowdsecurity/ssh-generic-test` dummy scenario by logging in to your server via SSH with a specific user. 1️⃣ Try to authenticate to your server via SSH using the following user: `crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl`. @@ -55,7 +55,7 @@ Note that this scenario has a delay of 5 minute before it can re-trigger _(black
- ☑️ Test CrowdSec WAF with an AppSec detection + ☑️ Test CrowdSec WAF with an AppSec detection If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test the detection of an AppSec scenario. It's similar to the HTTP detection test, but it will trigger `crowdsecurity/appsec-generic-test`. From bd1dccaed9a8b9b7d6e3d08da47fa64fedda7cc3 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 10:35:09 +0200 Subject: [PATCH 28/46] fkin docusaurus limitations :D we'll beautify it with react then --- .../getting_started/healthcheck.mdx | 21 +++++++++++-------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx index 2dd0c37a..3c4acb20 100644 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx @@ -27,7 +27,7 @@ The HTTP collection and the Linux collection contain **dummy scenarios** allowin :::
- Test **HTTP** detection + Test HTTP detection Let's trigger the `crowdsecurity/http-generic-test` dummy scenario by calling a *probe path* on your web server. 1️⃣ Request your service URL with the following path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` @@ -41,7 +41,7 @@ Note that this scenario has a delay of 5 minutes before it can re-trigger _(blac
- ☑️ Test **SSH** detection + Test SSH detection Let's trigger the `crowdsecurity/ssh-generic-test` dummy scenario by logging in to your server via SSH with a specific user. 1️⃣ Try to authenticate to your server via SSH using the following user: `crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl`. @@ -55,7 +55,7 @@ Note that this scenario has a delay of 5 minute before it can re-trigger _(black
- ☑️ Test CrowdSec WAF with an AppSec detection + Test CrowdSec WAF with an AppSec detection If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test the detection of an AppSec scenario. It's similar to the HTTP detection test, but it will trigger `crowdsecurity/appsec-generic-test`. @@ -84,7 +84,9 @@ If not, check the troubleshooting section below 🚨 Detection Troubleshooting *No alerts triggered? Let's investigate: Here are some tests to identify where the issue might be.* - If you installed CrowdSec Security Engine on the same host as the service you want to protect, the install wizards should have automatically detected the service and installed the appropriate parsers and scenarios. + + If you installed **CrowdSec Security Engine** on the same **Host** as the service you want to protect, the install wizards should have automatically detected the service and installed the appropriate parsers and scenarios. + If you have non-default paths or format for your logs, or if you chose other installation methods (docker, kubernetes..), you may need to manually install the parsers and scenarios. **This troubleshooting section will help you identify the issue and guide you through the necessary steps to fix it.** @@ -92,14 +94,13 @@ If not, check the troubleshooting section below
Acquisition metrics check: are your logs read and parsed? - The acquisition and parsing aspect of CrowdSec is crucial, as it tells The Security Engine which logs to read and how to parse them. - You can setup multiple datasources (files, syslog, etc.), for more details you can refer to the [datasources documentation](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro). + The acquisition and parsing aspect of CrowdSec is crucial, as it tells The Security Engine which logs to read and how to parse them. You can setup multiple datasources (files, syslog, etc.), for more details you can refer to the [datasources documentation](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro). Let's do a Top Down check using the `cscli metrics` command to see if your logs are being read and parsed correctly. sudo cscli metrics show acquisition parsers - - You should see the names of the log files/stream configured in the acquisition files, and the number of lines parsed for each of them. + - You should see the **names of the log files/stream** configured in the acquisition files, and the number of lines parsed for each of them. - The number of "Lines parsed" should be non-zero for each of the files you configured in the acquisition section. - The parsers metrics show you what parsers were successfully used. Look for the name of the parsers installed for the logs you're reading @@ -118,11 +119,13 @@ If not, check the troubleshooting section below
Acquisition Troubleshooting: are your logs properly declared as datasources - Your acquisition configuration is located in `/etc/crowdsec/acquis.yaml` or in `/etc/crowdsec/acquis.d/anyName.yaml`. + Your acquisition configuration is located in the `crowdsec` directory under `acquis.yaml` or `acquis.d/anyName.yaml`. + The crowdsec directory changes depending on your OS and installation method, for example, on Debian like OS it is usually located in `/etc/crowdsec/`. + + You may want to check your current acquisition configuration or [add a new acquisition file](https://doc.crowdsec.net/u/getting_started/post_installation/acquisition_new). - Check that the datasource is properly setup: file path or correct port for syslog server... - Ensure that the type declared matches the parser expected to be used: nginx, apache, syslog, etc. - for more information, check the [documentation for the various datasources](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro) -
From 9701207e7fe63c43da613c95c5b2b162bb3fcb67 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 13:05:24 +0200 Subject: [PATCH 29/46] ok summary jsut need space :D --- .../unversioned/getting_started/healthcheck.mdx | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx index 3c4acb20..76aa9003 100644 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/healthcheck.mdx @@ -27,7 +27,8 @@ The HTTP collection and the Linux collection contain **dummy scenarios** allowin :::
- Test HTTP detection + ☑️ Test **HTTP** detection + Let's trigger the `crowdsecurity/http-generic-test` dummy scenario by calling a *probe path* on your web server. 1️⃣ Request your service URL with the following path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` @@ -37,11 +38,15 @@ Let's trigger the `crowdsecurity/http-generic-test` dummy scenario by calling a sudo cscli alerts list -s crowdsecurity/http-generic-test -Note that this scenario has a delay of 5 minutes before it can re-trigger _(blackhole parameter of the scenario)_. +Notes: +- If it's done from a private IP the alert won't appear as private IPs are whitelisted by default. + - You can call the test url via a browser if you prefer, it might make it easier to test from an other device. +- This scenario has a delay of 5 minutes before it can re-trigger _(blackhole parameter of the scenario)_.
- Test SSH detection + ☑️ Test **SSH** detection + Let's trigger the `crowdsecurity/ssh-generic-test` dummy scenario by logging in to your server via SSH with a specific user. 1️⃣ Try to authenticate to your server via SSH using the following user: `crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl`. @@ -55,7 +60,8 @@ Note that this scenario has a delay of 5 minute before it can re-trigger _(black
- Test CrowdSec WAF with an AppSec detection + ☑️ Test CrowdSec WAF with an **AppSec** detection + If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test the detection of an AppSec scenario. It's similar to the HTTP detection test, but it will trigger `crowdsecurity/appsec-generic-test`. @@ -81,7 +87,7 @@ If so, you can proceed to the next phase of the health check: [**Connectivity ch If not, check the troubleshooting section below
- 🚨 Detection Troubleshooting + 🚨 **Detection Troubleshooting** *No alerts triggered? Let's investigate: Here are some tests to identify where the issue might be.* From 173c8cb9734d4e71b176c0d92ef299f9f407ebed Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 13:23:47 +0200 Subject: [PATCH 30/46] put healthcheck in side bar --- crowdsec-docs/sidebarsUnversioned.js | 1 + .../01_healthcheck.mdx} | 20 +++++++++---------- 2 files changed, 11 insertions(+), 10 deletions(-) rename crowdsec-docs/unversioned/getting_started/{healthcheck.mdx => post_installation/01_healthcheck.mdx} (95%) diff --git a/crowdsec-docs/sidebarsUnversioned.js b/crowdsec-docs/sidebarsUnversioned.js index aa319a40..bae51b58 100644 --- a/crowdsec-docs/sidebarsUnversioned.js +++ b/crowdsec-docs/sidebarsUnversioned.js @@ -577,6 +577,7 @@ module.exports = { id: "getting_started/next_steps", }, items: [ + "getting_started/post_installation/healthcheck", { type: "category", label: "CrowdSec Console", diff --git a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/01_healthcheck.mdx similarity index 95% rename from crowdsec-docs/unversioned/getting_started/healthcheck.mdx rename to crowdsec-docs/unversioned/getting_started/post_installation/01_healthcheck.mdx index 76aa9003..733718fb 100644 --- a/crowdsec-docs/unversioned/getting_started/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/01_healthcheck.mdx @@ -11,7 +11,7 @@ import HierarchicalList from '@site/src/components/HierarchicalList.js'; Welcome to the interactive health check of your CrowdSec setup. We'll have a Top-Down approach: First, check the proper functioning of the Security Engine's finality: -- **📡 Detecting** behaviors on your services. +- [**📡 Detecting**](#-detection-checks) behaviors on your services. - **🔗 Connectivity** with CrowdSec network to retrieve threats lists. - **🛡️ Protecting** your services by remediating alerts automatically with bouncers. @@ -27,7 +27,7 @@ The HTTP collection and the Linux collection contain **dummy scenarios** allowin :::
- ☑️ Test **HTTP** detection + 🌐 **HTTP** detection test Let's trigger the `crowdsecurity/http-generic-test` dummy scenario by calling a *probe path* on your web server. @@ -45,7 +45,7 @@ Notes:
- ☑️ Test **SSH** detection + 🔐 **SSH** detection test Let's trigger the `crowdsecurity/ssh-generic-test` dummy scenario by logging in to your server via SSH with a specific user. @@ -60,7 +60,7 @@ Note that this scenario has a delay of 5 minute before it can re-trigger _(black
- ☑️ Test CrowdSec WAF with an **AppSec** detection + 🛡️ **AppSec** detection test - CrowdSec WAF If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test the detection of an AppSec scenario. It's similar to the HTTP detection test, but it will trigger `crowdsecurity/appsec-generic-test`. @@ -111,18 +111,18 @@ If not, check the troubleshooting section below - The parsers metrics show you what parsers were successfully used. Look for the name of the parsers installed for the logs you're reading
- 🚨 If this command fails, check the [**CrowdSec Service Troubleshooting section**](#crowdsec_service_troubleshooting) + 🚨 If this command fails, check the [**CrowdSec Service Troubleshooting section**](#troubleshooting_service)
- 🚨 If you don't see the log names supposed to be parsed, check the [**Acquisition Troubleshooting section**](#acquisition_troubleshooting) + 🚨 If you don't see the log names supposed to be parsed, check the [**Acquisition Troubleshooting section**](#troubleshooting_acquisition)
- 🚨 If you don't see parsed lines, check the [**Collection Troubleshooting section****](#collection_troubleshooting) + 🚨 If you don't see parsed lines, check the [**Collection Troubleshooting section****](#troubleshooting_collection)
-
+
Acquisition Troubleshooting: are your logs properly declared as datasources Your acquisition configuration is located in the `crowdsec` directory under `acquis.yaml` or `acquis.d/anyName.yaml`. @@ -134,7 +134,7 @@ If not, check the troubleshooting section below - for more information, check the [documentation for the various datasources](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro)
-
+
Collection Troubleshooting: are the parsers and scenarios installed? #### Collection installation @@ -157,7 +157,7 @@ If not, check the troubleshooting section below
-
+
CrowdSec Service Troubleshooting: is the CrowdSec service running? From d24c23478fa59f78431a88a27bf47b069efdf7a0 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 13:28:01 +0200 Subject: [PATCH 31/46] fixed file name --- crowdsec-docs/sidebarsUnversioned.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crowdsec-docs/sidebarsUnversioned.js b/crowdsec-docs/sidebarsUnversioned.js index bae51b58..230f49ab 100644 --- a/crowdsec-docs/sidebarsUnversioned.js +++ b/crowdsec-docs/sidebarsUnversioned.js @@ -577,7 +577,7 @@ module.exports = { id: "getting_started/next_steps", }, items: [ - "getting_started/post_installation/healthcheck", + "getting_started/post_installation/01_healthcheck", { type: "category", label: "CrowdSec Console", From 1e95d22ea64613d2fa7cc6b62e51cc361291b8a2 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 13:30:10 +0200 Subject: [PATCH 32/46] fixed file name+ --- crowdsec-docs/sidebarsUnversioned.js | 2 +- .../post_installation/01_healthcheck.mdx | 215 ------------------ 2 files changed, 1 insertion(+), 216 deletions(-) delete mode 100644 crowdsec-docs/unversioned/getting_started/post_installation/01_healthcheck.mdx diff --git a/crowdsec-docs/sidebarsUnversioned.js b/crowdsec-docs/sidebarsUnversioned.js index 230f49ab..bae51b58 100644 --- a/crowdsec-docs/sidebarsUnversioned.js +++ b/crowdsec-docs/sidebarsUnversioned.js @@ -577,7 +577,7 @@ module.exports = { id: "getting_started/next_steps", }, items: [ - "getting_started/post_installation/01_healthcheck", + "getting_started/post_installation/healthcheck", { type: "category", label: "CrowdSec Console", diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/01_healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/01_healthcheck.mdx deleted file mode 100644 index 733718fb..00000000 --- a/crowdsec-docs/unversioned/getting_started/post_installation/01_healthcheck.mdx +++ /dev/null @@ -1,215 +0,0 @@ ---- -id: health_check -title: CrowdSec Security Engine Setup Health-Check ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import CodeBlock from '@theme/CodeBlock'; -import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; -import HierarchicalList from '@site/src/components/HierarchicalList.js'; - -Welcome to the interactive health check of your CrowdSec setup. We'll have a Top-Down approach: -First, check the proper functioning of the Security Engine's finality: -- [**📡 Detecting**](#-detection-checks) behaviors on your services. -- **🔗 Connectivity** with CrowdSec network to retrieve threats lists. -- **🛡️ Protecting** your services by remediating alerts automatically with bouncers. - -If the initial checks are not conclusive, we'll dive into a comprehensive troubleshooting for each section. - -## 📡 Detection checks - -### *Trigger CrowdSec's test scenarios* - -:::info -Check that your Security Engine properly reads and parses the logs of the services you protect. -The HTTP collection and the Linux collection contain **dummy scenarios** allowing you to prove your Security Engine works as intended, without having to do penetration tests, risking getting you banned from your own server. -::: - -
- 🌐 **HTTP** detection test - -Let's trigger the `crowdsecurity/http-generic-test` dummy scenario by calling a *probe path* on your web server. - -1️⃣ Request your service URL with the following path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` -curl -I https://\/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl - -2️⃣ You should see an alert for the scenario `crowdsecurity/http-generic-test` - -sudo cscli alerts list -s crowdsecurity/http-generic-test - -Notes: -- If it's done from a private IP the alert won't appear as private IPs are whitelisted by default. - - You can call the test url via a browser if you prefer, it might make it easier to test from an other device. -- This scenario has a delay of 5 minutes before it can re-trigger _(blackhole parameter of the scenario)_. -
- -
- 🔐 **SSH** detection test - -Let's trigger the `crowdsecurity/ssh-generic-test` dummy scenario by logging in to your server via SSH with a specific user. - -1️⃣ Try to authenticate to your server via SSH using the following user: `crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl`. -ssh crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl@\ - -2️⃣ You should see an alert for the scenario `crowdsecurity/ssh-generic-test` - -sudo cscli alerts list -s crowdsecurity/ssh-generic-test - -Note that this scenario has a delay of 5 minute before it can re-trigger _(blackhole parameter of the scenario)_. -
- -
- 🛡️ **AppSec** detection test - CrowdSec WAF - -If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test the detection of an AppSec scenario. -It's similar to the HTTP detection test, but it will trigger `crowdsecurity/appsec-generic-test`. - -Here is how to trigger the `crowdsecurity/appsec-generic-test` dummy scenario by calling a *probe path* on your web server. - -1️⃣ Request your service URL with the following path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` -curl -I http://your-service-url/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl - -2️⃣ You should see an alert for the scenario `crowdsecurity/appsec-generic-test` - -sudo cscli alerts list -s crowdsecurity/appsec-generic-test - -Note that this scenario has a delay of **1** minute before it can re-trigger _(blackhole parameter of the scenario)_. -
- -* * * - -### Were all the tests successful ? - -Were all the tests related to your setup successful ? -If so, you can proceed to the next phase of the health check: [**Connectivity checks**](#crowdsec-connectivity-checks). - -If not, check the troubleshooting section below - -
- 🚨 **Detection Troubleshooting** - - *No alerts triggered? Let's investigate: Here are some tests to identify where the issue might be.* - - If you installed **CrowdSec Security Engine** on the same **Host** as the service you want to protect, the install wizards should have automatically detected the service and installed the appropriate parsers and scenarios. - - If you have non-default paths or format for your logs, or if you chose other installation methods (docker, kubernetes..), you may need to manually install the parsers and scenarios. - - **This troubleshooting section will help you identify the issue and guide you through the necessary steps to fix it.** - -
- Acquisition metrics check: are your logs read and parsed? - - The acquisition and parsing aspect of CrowdSec is crucial, as it tells The Security Engine which logs to read and how to parse them. You can setup multiple datasources (files, syslog, etc.), for more details you can refer to the [datasources documentation](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro). - - Let's do a Top Down check using the `cscli metrics` command to see if your logs are being read and parsed correctly. - - sudo cscli metrics show acquisition parsers - - - You should see the **names of the log files/stream** configured in the acquisition files, and the number of lines parsed for each of them. - - The number of "Lines parsed" should be non-zero for each of the files you configured in the acquisition section. - - The parsers metrics show you what parsers were successfully used. Look for the name of the parsers installed for the logs you're reading - -
- 🚨 If this command fails, check the [**CrowdSec Service Troubleshooting section**](#troubleshooting_service) -
-
- 🚨 If you don't see the log names supposed to be parsed, check the [**Acquisition Troubleshooting section**](#troubleshooting_acquisition) -
-
- 🚨 If you don't see parsed lines, check the [**Collection Troubleshooting section****](#troubleshooting_collection) -
- -
- -
- Acquisition Troubleshooting: are your logs properly declared as datasources - - Your acquisition configuration is located in the `crowdsec` directory under `acquis.yaml` or `acquis.d/anyName.yaml`. - The crowdsec directory changes depending on your OS and installation method, for example, on Debian like OS it is usually located in `/etc/crowdsec/`. - - You may want to check your current acquisition configuration or [add a new acquisition file](https://doc.crowdsec.net/u/getting_started/post_installation/acquisition_new). - - Check that the datasource is properly setup: file path or correct port for syslog server... - - Ensure that the type declared matches the parser expected to be used: nginx, apache, syslog, etc. - - for more information, check the [documentation for the various datasources](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro) -
- -
- Collection Troubleshooting: are the parsers and scenarios installed? - - #### Collection installation - - On host installations your existing service should have been detected and appropriate collections (containing parsers and scenarios) installed automatically. - - How ever on other kind of installations, or in case of failed detection, you may need to install the collections manually. - - Check the installed collections with the command: - sudo cscli collections list - You can also list individual parsers and scenarios with: - sudo cscli parsers list - sudo cscli scenarios list - - If you see the parsers and scenarios related to your service, then you're good to go. - - - To install new collection, check the [CrowdSec Hub](https://hub.crowdsec.net/) for a collection named after your service (e.g., [nginx](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx), [apache](https://app.crowdsec.net/hub/author/crowdsecurity/collections/apache), [linux](https://app.crowdsec.net/hub/author/crowdsecurity/collections/linux), etc.). - - If you find the collection, install it using the command shown on the hub page and follow the instructions provided for the acquisition file. - - #### Log format mismatch - - If your logs are in a custom format, you may need to create your own parsers or modify the existing ones to match your log format. - - You can find more information on how to create your own parsers in the [CrowdSec documentation](https://doc.crowdsec.net/docs/next/log_processor/parsers/format). - -
- -
- CrowdSec Service Troubleshooting: is the CrowdSec service running? - - - -
-
- -## CrowdSec Connectivity checks - -Check that your Security Engine can communicate with the CrowdSec Central API and receive the community blocklist and additional ones. - -// Do we make them check console enroll here? or just cscli capi status? - *Recommend console enrollment here; the previous checks establish core functionality. Enrollment becomes the next logical step.* - -```bash -sudo cscli capi status -``` -- Should show: "You can successfully interact with Central API (CAPI)" -- If this test fail, try to enrol to the console to check what's going on. - -
- Network Connectivity Troubleshooting - -* online api credentials exist in `/etc/crowdsec/config/online_api_credentials.yaml` (default path) -* Firewall rules blocking outbound connections to the CrowdSec Central API. -* DNS resolution issues. -* Proxy server configuration. -* Connectivity issues within Docker containers. - -
- -## Remediation checks - -* Canary tester - -(This section requires a robust, safe, and automated canary tester solution. The biggest challenge is preventing users from accidentally banning themselves. This section can be skipped on V0, as you mentioned, and focused on later. For V1, It could be a URL to a third party machine for testing, or a private machine dedicated to running a canary test) - -
- Bouncer Troubleshooting - -* Verifying that the bouncer is registered and active. -* Checking bouncer logs for errors. -* Troubleshooting communication issues between the bouncer and the CrowdSec LAPI. -* Firewall configuration for the bouncer. -* Specific troubleshooting steps for common bouncers (firewall-bouncer, nginx-bouncer, etc.).) - -
- -
- Profile Troubleshooting: Steps - -* Verifying profile configuration. -* Troubleshooting errors related to profile application. -* Understanding the impact of profiles on detection and remediation.) - -
\ No newline at end of file From 7f4ebaa022e380e46321715f2ddf2e417f0384bb Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 13:30:23 +0200 Subject: [PATCH 33/46] fixed file name+ --- .../post_installation/healthcheck.mdx | 215 ++++++++++++++++++ 1 file changed, 215 insertions(+) create mode 100644 crowdsec-docs/unversioned/getting_started/post_installation/healthcheck.mdx diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/healthcheck.mdx new file mode 100644 index 00000000..733718fb --- /dev/null +++ b/crowdsec-docs/unversioned/getting_started/post_installation/healthcheck.mdx @@ -0,0 +1,215 @@ +--- +id: health_check +title: CrowdSec Security Engine Setup Health-Check +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; +import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; +import HierarchicalList from '@site/src/components/HierarchicalList.js'; + +Welcome to the interactive health check of your CrowdSec setup. We'll have a Top-Down approach: +First, check the proper functioning of the Security Engine's finality: +- [**📡 Detecting**](#-detection-checks) behaviors on your services. +- **🔗 Connectivity** with CrowdSec network to retrieve threats lists. +- **🛡️ Protecting** your services by remediating alerts automatically with bouncers. + +If the initial checks are not conclusive, we'll dive into a comprehensive troubleshooting for each section. + +## 📡 Detection checks + +### *Trigger CrowdSec's test scenarios* + +:::info +Check that your Security Engine properly reads and parses the logs of the services you protect. +The HTTP collection and the Linux collection contain **dummy scenarios** allowing you to prove your Security Engine works as intended, without having to do penetration tests, risking getting you banned from your own server. +::: + +
+ 🌐 **HTTP** detection test + +Let's trigger the `crowdsecurity/http-generic-test` dummy scenario by calling a *probe path* on your web server. + +1️⃣ Request your service URL with the following path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` +curl -I https://\/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl + +2️⃣ You should see an alert for the scenario `crowdsecurity/http-generic-test` + +sudo cscli alerts list -s crowdsecurity/http-generic-test + +Notes: +- If it's done from a private IP the alert won't appear as private IPs are whitelisted by default. + - You can call the test url via a browser if you prefer, it might make it easier to test from an other device. +- This scenario has a delay of 5 minutes before it can re-trigger _(blackhole parameter of the scenario)_. +
+ +
+ 🔐 **SSH** detection test + +Let's trigger the `crowdsecurity/ssh-generic-test` dummy scenario by logging in to your server via SSH with a specific user. + +1️⃣ Try to authenticate to your server via SSH using the following user: `crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl`. +ssh crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl@\ + +2️⃣ You should see an alert for the scenario `crowdsecurity/ssh-generic-test` + +sudo cscli alerts list -s crowdsecurity/ssh-generic-test + +Note that this scenario has a delay of 5 minute before it can re-trigger _(blackhole parameter of the scenario)_. +
+ +
+ 🛡️ **AppSec** detection test - CrowdSec WAF + +If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test the detection of an AppSec scenario. +It's similar to the HTTP detection test, but it will trigger `crowdsecurity/appsec-generic-test`. + +Here is how to trigger the `crowdsecurity/appsec-generic-test` dummy scenario by calling a *probe path* on your web server. + +1️⃣ Request your service URL with the following path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` +curl -I http://your-service-url/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl + +2️⃣ You should see an alert for the scenario `crowdsecurity/appsec-generic-test` + +sudo cscli alerts list -s crowdsecurity/appsec-generic-test + +Note that this scenario has a delay of **1** minute before it can re-trigger _(blackhole parameter of the scenario)_. +
+ +* * * + +### Were all the tests successful ? + +Were all the tests related to your setup successful ? +If so, you can proceed to the next phase of the health check: [**Connectivity checks**](#crowdsec-connectivity-checks). + +If not, check the troubleshooting section below + +
+ 🚨 **Detection Troubleshooting** + + *No alerts triggered? Let's investigate: Here are some tests to identify where the issue might be.* + + If you installed **CrowdSec Security Engine** on the same **Host** as the service you want to protect, the install wizards should have automatically detected the service and installed the appropriate parsers and scenarios. + + If you have non-default paths or format for your logs, or if you chose other installation methods (docker, kubernetes..), you may need to manually install the parsers and scenarios. + + **This troubleshooting section will help you identify the issue and guide you through the necessary steps to fix it.** + +
+ Acquisition metrics check: are your logs read and parsed? + + The acquisition and parsing aspect of CrowdSec is crucial, as it tells The Security Engine which logs to read and how to parse them. You can setup multiple datasources (files, syslog, etc.), for more details you can refer to the [datasources documentation](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro). + + Let's do a Top Down check using the `cscli metrics` command to see if your logs are being read and parsed correctly. + + sudo cscli metrics show acquisition parsers + + - You should see the **names of the log files/stream** configured in the acquisition files, and the number of lines parsed for each of them. + - The number of "Lines parsed" should be non-zero for each of the files you configured in the acquisition section. + - The parsers metrics show you what parsers were successfully used. Look for the name of the parsers installed for the logs you're reading + +
+ 🚨 If this command fails, check the [**CrowdSec Service Troubleshooting section**](#troubleshooting_service) +
+
+ 🚨 If you don't see the log names supposed to be parsed, check the [**Acquisition Troubleshooting section**](#troubleshooting_acquisition) +
+
+ 🚨 If you don't see parsed lines, check the [**Collection Troubleshooting section****](#troubleshooting_collection) +
+ +
+ +
+ Acquisition Troubleshooting: are your logs properly declared as datasources + + Your acquisition configuration is located in the `crowdsec` directory under `acquis.yaml` or `acquis.d/anyName.yaml`. + The crowdsec directory changes depending on your OS and installation method, for example, on Debian like OS it is usually located in `/etc/crowdsec/`. + + You may want to check your current acquisition configuration or [add a new acquisition file](https://doc.crowdsec.net/u/getting_started/post_installation/acquisition_new). + - Check that the datasource is properly setup: file path or correct port for syslog server... + - Ensure that the type declared matches the parser expected to be used: nginx, apache, syslog, etc. + - for more information, check the [documentation for the various datasources](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro) +
+ +
+ Collection Troubleshooting: are the parsers and scenarios installed? + + #### Collection installation + - On host installations your existing service should have been detected and appropriate collections (containing parsers and scenarios) installed automatically. + - How ever on other kind of installations, or in case of failed detection, you may need to install the collections manually. + + Check the installed collections with the command: + sudo cscli collections list + You can also list individual parsers and scenarios with: + sudo cscli parsers list + sudo cscli scenarios list + - If you see the parsers and scenarios related to your service, then you're good to go. + + - To install new collection, check the [CrowdSec Hub](https://hub.crowdsec.net/) for a collection named after your service (e.g., [nginx](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx), [apache](https://app.crowdsec.net/hub/author/crowdsecurity/collections/apache), [linux](https://app.crowdsec.net/hub/author/crowdsecurity/collections/linux), etc.). + - If you find the collection, install it using the command shown on the hub page and follow the instructions provided for the acquisition file. + + #### Log format mismatch + - If your logs are in a custom format, you may need to create your own parsers or modify the existing ones to match your log format. + - You can find more information on how to create your own parsers in the [CrowdSec documentation](https://doc.crowdsec.net/docs/next/log_processor/parsers/format). + +
+ +
+ CrowdSec Service Troubleshooting: is the CrowdSec service running? + + + +
+
+ +## CrowdSec Connectivity checks + +Check that your Security Engine can communicate with the CrowdSec Central API and receive the community blocklist and additional ones. + +// Do we make them check console enroll here? or just cscli capi status? - *Recommend console enrollment here; the previous checks establish core functionality. Enrollment becomes the next logical step.* + +```bash +sudo cscli capi status +``` +- Should show: "You can successfully interact with Central API (CAPI)" +- If this test fail, try to enrol to the console to check what's going on. + +
+ Network Connectivity Troubleshooting + +* online api credentials exist in `/etc/crowdsec/config/online_api_credentials.yaml` (default path) +* Firewall rules blocking outbound connections to the CrowdSec Central API. +* DNS resolution issues. +* Proxy server configuration. +* Connectivity issues within Docker containers. + +
+ +## Remediation checks + +* Canary tester + +(This section requires a robust, safe, and automated canary tester solution. The biggest challenge is preventing users from accidentally banning themselves. This section can be skipped on V0, as you mentioned, and focused on later. For V1, It could be a URL to a third party machine for testing, or a private machine dedicated to running a canary test) + +
+ Bouncer Troubleshooting + +* Verifying that the bouncer is registered and active. +* Checking bouncer logs for errors. +* Troubleshooting communication issues between the bouncer and the CrowdSec LAPI. +* Firewall configuration for the bouncer. +* Specific troubleshooting steps for common bouncers (firewall-bouncer, nginx-bouncer, etc.).) + +
+ +
+ Profile Troubleshooting: Steps + +* Verifying profile configuration. +* Troubleshooting errors related to profile application. +* Understanding the impact of profiles on detection and remediation.) + +
\ No newline at end of file From af82fa5dc8b4b72b33d5a749e33310a4d374fd33 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 14:02:57 +0200 Subject: [PATCH 34/46] fixed naming vs id + added remediation check content --- crowdsec-docs/sidebarsUnversioned.js | 2 +- .../{healthcheck.mdx => health_check.mdx} | 70 +++++++++++++++---- 2 files changed, 56 insertions(+), 16 deletions(-) rename crowdsec-docs/unversioned/getting_started/post_installation/{healthcheck.mdx => health_check.mdx} (78%) diff --git a/crowdsec-docs/sidebarsUnversioned.js b/crowdsec-docs/sidebarsUnversioned.js index bae51b58..8d3e6def 100644 --- a/crowdsec-docs/sidebarsUnversioned.js +++ b/crowdsec-docs/sidebarsUnversioned.js @@ -577,7 +577,7 @@ module.exports = { id: "getting_started/next_steps", }, items: [ - "getting_started/post_installation/healthcheck", + "getting_started/post_installation/health_check", { type: "category", label: "CrowdSec Console", diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/healthcheck.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx similarity index 78% rename from crowdsec-docs/unversioned/getting_started/post_installation/healthcheck.mdx rename to crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index 733718fb..72ae2188 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/healthcheck.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -165,7 +165,7 @@ If not, check the troubleshooting section below
-## CrowdSec Connectivity checks +## 🔌 CrowdSec Connectivity checks Check that your Security Engine can communicate with the CrowdSec Central API and receive the community blocklist and additional ones. @@ -188,28 +188,68 @@ sudo cscli capi status
-## Remediation checks +## ✋🏻 Remediation checks -* Canary tester +Now that we've confirmed that the Security Engine can detect threats, let's verify that your bouncers are correctly applying decisions and blocking malicious IPs. -(This section requires a robust, safe, and automated canary tester solution. The biggest challenge is preventing users from accidentally banning themselves. This section can be skipped on V0, as you mentioned, and focused on later. For V1, It could be a URL to a third party machine for testing, or a private machine dedicated to running a canary test) +::info A note on future improvements +In the future, this health-check will include an option to trigger a remote test from the CrowdSec Console, which will safely probe your defenses from an external IP. +For now, we will perform a manual test by temporarily banning your own IP address. Please read the following warning carefully before proceeding. +:::
- Bouncer Troubleshooting - -* Verifying that the bouncer is registered and active. -* Checking bouncer logs for errors. -* Troubleshooting communication issues between the bouncer and the CrowdSec LAPI. -* Firewall configuration for the bouncer. -* Specific troubleshooting steps for common bouncers (firewall-bouncer, nginx-bouncer, etc.).) + ✋🏻 Bouncer Remediation test + +This test involves manually creating a ban decision against your own public IP address for a very short period (1 minute). + +::danger BE CAREFUL: Risk of Self-Lockout +This procedure will temporarily block your access to the services protected by your bouncer. +Make sure to properly set the TTL to a low expiration time (1 minute). OR do it from a device with a different public IP address than the client you're using to setup CrowdSec. +:: + +1️⃣ Find out you public address by using the following command: +curl api.ipify.org +or +curl curl ipinfo.io/ip + +2️⃣ Create a ban decision for your public IP address with a TTL of 1 minute: +sudo cscli decisions add ban --ip \ --duration 1m --reason "CrowdSec remediation test" + +3️⃣ Try to access your service from the same public IP address. You should be blocked by the bouncer. + +4️⃣ Wait for 1 minute, then check the decisions list to see if the decision has been removed
+### Were all the tests successful ? + +If you were successfully blocked, congratulations! Your remediation setup is working correctly. 🎉 +You might want to continue to the next recommended steps: +- Enroll your Security Engine to the [CrowdSec Console](/u/getting_started/post_installation/console/) +- Then subscribe to more blocklists to benefit from additionnl proactive prevention +
- Profile Troubleshooting: Steps + 🚨 **Remediation Troubleshooting** + + ... -* Verifying profile configuration. -* Troubleshooting errors related to profile application. -* Understanding the impact of profiles on detection and remediation.) +
+ Bouncer Configuration Troubleshooting + + * Verifying that the bouncer is registered and active. + * Checking bouncer logs for errors. + * Troubleshooting communication issues between the bouncer and the CrowdSec LAPI. + * Firewall configuration for the bouncer. + * Specific troubleshooting steps for common bouncers (firewall-bouncer, nginx-bouncer, etc.).) + +
+
+ Profile Troubleshooting + + * Verifying profile configuration. + * Troubleshooting errors related to profile application. + * Understanding the impact of profiles on detection and remediation.) + +
\ No newline at end of file From 50fe29099b1a681c31b2ad0548797af8bd707ff3 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 16:03:43 +0200 Subject: [PATCH 35/46] connectivity update --- .../post_installation/health_check.mdx | 39 ++++++++++++------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index 72ae2188..16e8e9b8 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -167,24 +167,35 @@ If not, check the troubleshooting section below ## 🔌 CrowdSec Connectivity checks -Check that your Security Engine can communicate with the CrowdSec Central API and receive the community blocklist and additional ones. +Let's check that your Security Engine can communicate with the CrowdSec Central API (CAPI). This connection is essential for participating in the community and protecting your setup with the collective intelligence of the network. +The CAPI allows your instance to receive a curated blocklist of malicious IP addresses identified and verified by the entire CrowdSec network. This proactive defense, known as the Community Blocklist, protects you from aggressors before they even target your services. It also allows your Security Engine to report the malicious IPs it detects, contributing to the strength of the community. -// Do we make them check console enroll here? or just cscli capi status? - *Recommend console enrollment here; the previous checks establish core functionality. Enrollment becomes the next logical step.* +
+ 🔌 CrowdSec Central API connectivity -```bash -sudo cscli capi status -``` -- Should show: "You can successfully interact with Central API (CAPI)" -- If this test fail, try to enrol to the console to check what's going on. + The most direct way to verify connectivity is to see if your instance has already received decisions from the community blocklist. -
- Network Connectivity Troubleshooting + 1️⃣ List decisions coming from CAPI + + sudo cscli decisions list --origin CAPI -* online api credentials exist in `/etc/crowdsec/config/online_api_credentials.yaml` (default path) -* Firewall rules blocking outbound connections to the CrowdSec Central API. -* DNS resolution issues. -* Proxy server configuration. -* Connectivity issues within Docker containers. + 2️⃣ If you see decisions listed, it means your Security Engine is successfully connected to the CAPI and receiving decisions. +
+ +
+ 🚨 Connectivity Troubleshooting + + Check CAPI status with the command: + sudo cscli capi status + - Should show: "You can successfully interact with Central API (CAPI)" + - If this test fail, try to enrol to the console to check what's going on. + + Other common issues that can prevent your Security Engine from connecting to the CAPI include: + - online api credentials exist in `/etc/crowdsec/config/online_api_credentials.yaml` (default path) + - Firewall rules blocking outbound connections to the CrowdSec Central API. + - DNS resolution issues. + - Proxy server configuration. + - Connectivity issues within Docker containers.
From 5080cfcc5c76cf17404ab8753c72192a0ce37ecf Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 17:03:06 +0200 Subject: [PATCH 36/46] mini clean and content+ --- .../getting_started/post_installation/health_check.mdx | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index 16e8e9b8..c0ddd24b 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -160,13 +160,13 @@ If not, check the troubleshooting section below
CrowdSec Service Troubleshooting: is the CrowdSec service running? - -
## 🔌 CrowdSec Connectivity checks +### *List the community-blocked IPs* + Let's check that your Security Engine can communicate with the CrowdSec Central API (CAPI). This connection is essential for participating in the community and protecting your setup with the collective intelligence of the network. The CAPI allows your instance to receive a curated blocklist of malicious IP addresses identified and verified by the entire CrowdSec network. This proactive defense, known as the Community Blocklist, protects you from aggressors before they even target your services. It also allows your Security Engine to report the malicious IPs it detects, contributing to the strength of the community. @@ -182,6 +182,8 @@ The CAPI allows your instance to receive a curated blocklist of malicious IP add 2️⃣ If you see decisions listed, it means your Security Engine is successfully connected to the CAPI and receiving decisions.
+### Were all the tests successful ? +
🚨 Connectivity Troubleshooting @@ -201,6 +203,8 @@ The CAPI allows your instance to receive a curated blocklist of malicious IP add ## ✋🏻 Remediation checks +### *Validate Blocks or Captchas* + Now that we've confirmed that the Security Engine can detect threats, let's verify that your bouncers are correctly applying decisions and blocking malicious IPs. ::info A note on future improvements @@ -235,6 +239,7 @@ or ### Were all the tests successful ? If you were successfully blocked, congratulations! Your remediation setup is working correctly. 🎉 + You might want to continue to the next recommended steps: - Enroll your Security Engine to the [CrowdSec Console](/u/getting_started/post_installation/console/) - Then subscribe to more blocklists to benefit from additionnl proactive prevention From cb62dffb4a65befcb09e24cb429725ff3900156b Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 17:16:52 +0200 Subject: [PATCH 37/46] mini clean and content+ --- .../post_installation/health_check.mdx | 23 +++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index c0ddd24b..18d75917 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -159,6 +159,22 @@ If not, check the troubleshooting section below
CrowdSec Service Troubleshooting: is the CrowdSec service running? + Make sure the CrowdSec service is running properly. You can check the status of the service with the following command: + sudo systemctl status crowdsec + - You should see the service status as "active (running)". + + If the service is not running, you can start it with the following command: + sudo systemctl start crowdsec + + If the service fails to start, you can check the logs for more information: + sudo journalctl -u crowdsec + + Reasons for the service not running can include: + - Misconfiguration in the `config.yaml` file. + - Port conflicts with other services. By default, CrowdSec uses port 8080 for the Local API (LAPI) and port 6060 for the Local API metrics. + - The port configuration can be setup in `config.yaml` file or by setting the environment variables depending on your implementation + - Insufficient permissions to access the log files or directories. + - Acquisition files format errors.
@@ -193,7 +209,10 @@ The CAPI allows your instance to receive a curated blocklist of malicious IP add - If this test fail, try to enrol to the console to check what's going on. Other common issues that can prevent your Security Engine from connecting to the CAPI include: - - online api credentials exist in `/etc/crowdsec/config/online_api_credentials.yaml` (default path) + - Check that the online api credentials `online_api_credentials.yaml` exist in your crowdsec config directory + - If they don't exist, you can create them by running the command: + sudo cscli capi register + - Check that the `crowdsec` service is running and properly configured to connect to the CAPI. - Firewall rules blocking outbound connections to the CrowdSec Central API. - DNS resolution issues. - Proxy server configuration. @@ -214,7 +233,7 @@ For now, we will perform a manual test by temporarily banning your own IP addres
✋🏻 Bouncer Remediation test - + This test involves manually creating a ban decision against your own public IP address for a very short period (1 minute). ::danger BE CAREFUL: Risk of Self-Lockout From c035e7c93e00bfc725d02129f447de313cd94b8f Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 21:37:08 +0200 Subject: [PATCH 38/46] ok, need to read from the start now --- .../post_installation/health_check.mdx | 31 +++++++++++++------ 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index 18d75917..250d451a 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -9,13 +9,20 @@ import CodeBlock from '@theme/CodeBlock'; import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; import HierarchicalList from '@site/src/components/HierarchicalList.js'; -Welcome to the interactive health check of your CrowdSec setup. We'll have a Top-Down approach: -First, check the proper functioning of the Security Engine's finality: +Health Check Version: 0.1.0 + +Welcome to the interactive health check of your CrowdSec setup. +We'll guide you through a series of tests to ensure that your Security Stack is fully functional and ready to protect your services: **Detecting**, **Threat Sharing** and **Remediating**. +This guide covers cases of protecting common services such as web servers (HTTP) and SSH. + +Via a **top-down approach** we'll test the end goal of components, and then dive into detailed troubleshooting if needed. + +This health check is divided into three main sections: - [**📡 Detecting**](#-detection-checks) behaviors on your services. -- **🔗 Connectivity** with CrowdSec network to retrieve threats lists. -- **🛡️ Protecting** your services by remediating alerts automatically with bouncers. +- [**🔗 Connectivity**](#-crowdsec-connectivity-checks) with CrowdSec network to retrieve the community blocklist. +- [**🛡️ Protecting**](#-remediation-checks) your services by remediating alerts automatically with bouncers. -If the initial checks are not conclusive, we'll dive into a comprehensive troubleshooting for each section. +* * * ## 📡 Detection checks @@ -50,7 +57,7 @@ Notes: Let's trigger the `crowdsecurity/ssh-generic-test` dummy scenario by logging in to your server via SSH with a specific user. 1️⃣ Try to authenticate to your server via SSH using the following user: `crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl`. -ssh crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl@\ +ssh crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl@\ 2️⃣ You should see an alert for the scenario `crowdsecurity/ssh-generic-test` @@ -98,7 +105,7 @@ If not, check the troubleshooting section below **This troubleshooting section will help you identify the issue and guide you through the necessary steps to fix it.**
- Acquisition metrics check: are your logs read and parsed? + Are your logs being properly read and parsed? The acquisition and parsing aspect of CrowdSec is crucial, as it tells The Security Engine which logs to read and how to parse them. You can setup multiple datasources (files, syslog, etc.), for more details you can refer to the [datasources documentation](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro). @@ -126,7 +133,7 @@ If not, check the troubleshooting section below Acquisition Troubleshooting: are your logs properly declared as datasources Your acquisition configuration is located in the `crowdsec` directory under `acquis.yaml` or `acquis.d/anyName.yaml`. - The crowdsec directory changes depending on your OS and installation method, for example, on Debian like OS it is usually located in `/etc/crowdsec/`. + The CrowdSec directory changes depending on your OS and installation method, for example, on Debian like OS it is usually located in `/etc/crowdsec/`. You may want to check your current acquisition configuration or [add a new acquisition file](https://doc.crowdsec.net/u/getting_started/post_installation/acquisition_new). - Check that the datasource is properly setup: file path or correct port for syslog server... @@ -249,7 +256,7 @@ or 2️⃣ Create a ban decision for your public IP address with a TTL of 1 minute: sudo cscli decisions add ban --ip \ --duration 1m --reason "CrowdSec remediation test" -3️⃣ Try to access your service from the same public IP address. You should be blocked by the bouncer. +3️⃣ Try to access your service (website, api ...) from the same public IP address. \> You should be blocked by the bouncer. 4️⃣ Wait for 1 minute, then check the decisions list to see if the decision has been removed @@ -287,4 +294,8 @@ You might want to continue to the next recommended steps: * Understanding the impact of profiles on detection and remediation.)
-
\ No newline at end of file +
+ +## Your feedback is important! + +If you have any suggestions or improvements for this health check, please let us know by opening an issue on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs/issues/new) or on our [discord](https://discord.gg/wGN7ShmEE8) \ No newline at end of file From 07dd181174495e3126b9b5d5dc86c8c0749ed4e5 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 22:39:42 +0200 Subject: [PATCH 39/46] improving detection section 1/2 --- crowdsec-docs/sidebarsUnversioned.js | 6 +- .../post_installation/health_check.mdx | 106 +++++++++--------- 2 files changed, 60 insertions(+), 52 deletions(-) diff --git a/crowdsec-docs/sidebarsUnversioned.js b/crowdsec-docs/sidebarsUnversioned.js index 8d3e6def..83cb68aa 100644 --- a/crowdsec-docs/sidebarsUnversioned.js +++ b/crowdsec-docs/sidebarsUnversioned.js @@ -577,7 +577,11 @@ module.exports = { id: "getting_started/next_steps", }, items: [ - "getting_started/post_installation/health_check", + { + type: "doc", + id: "getting_started/post_installation/health_check", + label: "Stack Health-Check", + }, { type: "category", label: "CrowdSec Console", diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index 250d451a..3d0e2c6d 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -9,18 +9,19 @@ import CodeBlock from '@theme/CodeBlock'; import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; import HierarchicalList from '@site/src/components/HierarchicalList.js'; -Health Check Version: 0.1.0 +Health Check Version: 0.1.0 -Welcome to the interactive health check of your CrowdSec setup. -We'll guide you through a series of tests to ensure that your Security Stack is fully functional and ready to protect your services: **Detecting**, **Threat Sharing** and **Remediating**. -This guide covers cases of protecting common services such as web servers (HTTP) and SSH. +Welcome to the interactive Health-Check of your CrowdSec setup. +We'll guide you through a series of tests to ensure that your Security Stack is fully functional and ready to protect your services: +**Detecting**, **Threat Sharing** and **Remediating**. +*This guide covers cases of protecting common services such as web servers (HTTP) and SSH.* -Via a **top-down approach** we'll test the end goal of components, and then dive into detailed troubleshooting if needed. +We'll first test the final functionality of each component (top-down approach) before diving into detailed troubleshooting if issues arise. This health check is divided into three main sections: -- [**📡 Detecting**](#-detection-checks) behaviors on your services. -- [**🔗 Connectivity**](#-crowdsec-connectivity-checks) with CrowdSec network to retrieve the community blocklist. -- [**🛡️ Protecting**](#-remediation-checks) your services by remediating alerts automatically with bouncers. +- [**📡 Detection**](#-detection-checks): Ensuring CrowdSec properly detects threats targeting your services. +- [**🔗 Connectivity**](#-crowdsec-connectivity-checks): Verifying communication with the CrowdSec network to receive the community blocklist. +- [**🛡️ Protection**](#-remediation-checks): Confirming that your bouncers automatically block threats detected by CrowdSec * * * @@ -28,103 +29,106 @@ This health check is divided into three main sections: ### *Trigger CrowdSec's test scenarios* -:::info -Check that your Security Engine properly reads and parses the logs of the services you protect. -The HTTP collection and the Linux collection contain **dummy scenarios** allowing you to prove your Security Engine works as intended, without having to do penetration tests, risking getting you banned from your own server. -::: +Let's use CrowdSec's built-in **dummy scenarios** (HTTP and Linux) to safely verify your Security Engine detects threats, without risking accidental self-blocking.
🌐 **HTTP** detection test -Let's trigger the `crowdsecurity/http-generic-test` dummy scenario by calling a *probe path* on your web server. +We'll trigger the dummy scenario `crowdsecurity/http-generic-test` by accessing a **probe path** on your web server. -1️⃣ Request your service URL with the following path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` +1️⃣ Access your service URL with this path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` curl -I https://\/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl -2️⃣ You should see an alert for the scenario `crowdsecurity/http-generic-test` - +2️⃣ Confirm the alert has triggered for the scenario `crowdsecurity/http-generic-test` sudo cscli alerts list -s crowdsecurity/http-generic-test -Notes: -- If it's done from a private IP the alert won't appear as private IPs are whitelisted by default. - - You can call the test url via a browser if you prefer, it might make it easier to test from an other device. -- This scenario has a delay of 5 minutes before it can re-trigger _(blackhole parameter of the scenario)_. +**Notes:** +- Requests from private IP addresses won't trigger alerts (private IPs are whitelisted by default). + - You can also test via a browser if easier, especially from another device. +- This scenario can be triggered again only after a 5-minutes delay.
🔐 **SSH** detection test -Let's trigger the `crowdsecurity/ssh-generic-test` dummy scenario by logging in to your server via SSH with a specific user. +We'll trigger the dummy scenario `crowdsecurity/ssh-generic-test` by attempting an SSH login with a specific username. -1️⃣ Try to authenticate to your server via SSH using the following user: `crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl`. +1️⃣ Attempt SSH login using this username: `crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl`. ssh crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl@\ -2️⃣ You should see an alert for the scenario `crowdsecurity/ssh-generic-test` - +2️⃣ Confirm the alert has triggered for the scenario `crowdsecurity/ssh-generic-test` sudo cscli alerts list -s crowdsecurity/ssh-generic-test -Note that this scenario has a delay of 5 minute before it can re-trigger _(blackhole parameter of the scenario)_. +**Notes:** +- This scenario can only be triggered again after a 5-minutes delay.
🛡️ **AppSec** detection test - CrowdSec WAF -If you are using an AppSec-capable bouncer and have configured CrowdSec WAF, you can test the detection of an AppSec scenario. -It's similar to the HTTP detection test, but it will trigger `crowdsecurity/appsec-generic-test`. +If you've enabled an AppSec-capable bouncer with CrowdSec WAF, you can trigger the `crowdsecurity/appsec-generic-test` dummy scenario. +It would have triggered along with the HTTP detection test, but it is worth mentioning here as well. Here is how to trigger the `crowdsecurity/appsec-generic-test` dummy scenario by calling a *probe path* on your web server. -1️⃣ Request your service URL with the following path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` -curl -I http://your-service-url/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl +We'll trigger the dummy scenario `crowdsecurity/appsec-generic-test` by accessing a **probe path** on your web server. -2️⃣ You should see an alert for the scenario `crowdsecurity/appsec-generic-test` +1️⃣ Access your service URL with this path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` +curl -I https://\/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl +2️⃣ Confirm the alert has triggered for the scenario `crowdsecurity/appsec-generic-test` sudo cscli alerts list -s crowdsecurity/appsec-generic-test -Note that this scenario has a delay of **1** minute before it can re-trigger _(blackhole parameter of the scenario)_. +**Notes:** +- This scenario can only be triggered again after a 1-minute delay.
* * * ### Were all the tests successful ? -Were all the tests related to your setup successful ? -If so, you can proceed to the next phase of the health check: [**Connectivity checks**](#crowdsec-connectivity-checks). +Were all the tests related to your setup successful? +👍 If so, you can proceed to the next phase of the health check: [**Connectivity checks**](#-crowdsec-connectivity-checks). -If not, check the troubleshooting section below +🛠️ If not, check the troubleshooting section below.
- 🚨 **Detection Troubleshooting** + 🐞 **Detection Troubleshooting** - *No alerts triggered? Let's investigate: Here are some tests to identify where the issue might be.* + **No alert triggered? Let's find out why.** - If you installed **CrowdSec Security Engine** on the same **Host** as the service you want to protect, the install wizards should have automatically detected the service and installed the appropriate parsers and scenarios. - - If you have non-default paths or format for your logs, or if you chose other installation methods (docker, kubernetes..), you may need to manually install the parsers and scenarios. + If you installed CrowdSec on the same **host** as the service you're protecting, it should have auto-detected it and installed the right collections of parsers and scenarios. + However, if you're using *custom log paths*, *unusual log formats*, or running in *Docker/Kubernetes*, you might need to configure some things manually. - **This troubleshooting section will help you identify the issue and guide you through the necessary steps to fix it.** + **This section will help you pinpoint the issue and walk you through how to fix it.**
Are your logs being properly read and parsed? - The acquisition and parsing aspect of CrowdSec is crucial, as it tells The Security Engine which logs to read and how to parse them. You can setup multiple datasources (files, syslog, etc.), for more details you can refer to the [datasources documentation](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro). - - Let's do a Top Down check using the `cscli metrics` command to see if your logs are being read and parsed correctly. + CrowdSec needs to know what logs to read and how to interpret them. + This is handled by the acquisition configuration (log sources) and parsing (how to read them). + Multiple log sources can be defined in the acquisition(s) configuration files and they support diverse datasources (files, syslog, etc.). + For more details you can refer to the [datasources documentation](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro). + We'll look at the security engine **metrics** to see if logs are **being read** and if what's read is **parsed correctly**. + We'll do that using the `cscli metrics` command: sudo cscli metrics show acquisition parsers - - You should see the **names of the log files/stream** configured in the acquisition files, and the number of lines parsed for each of them. - - The number of "Lines parsed" should be non-zero for each of the files you configured in the acquisition section. - - The parsers metrics show you what parsers were successfully used. Look for the name of the parsers installed for the logs you're reading + Under **Acquisition Metrics** you should see: + - The source name of the log files or streams that have been read and the number of lines read and parsed for each of them. + - If you don't see any sources or some you have configured are missing, it means that the acquisition configuration is not properly set up. + - A non zero number of "Lines parsed" is expected for each source, proving that the appropriate parser was found and used. + + Under The **Parsers Metrics** you have the details of the parsers used.
- 🚨 If this command fails, check the [**CrowdSec Service Troubleshooting section**](#troubleshooting_service) + 🐞 If this command fails, check the [**CrowdSec Service Troubleshooting section**](#troubleshooting_service)
- 🚨 If you don't see the log names supposed to be parsed, check the [**Acquisition Troubleshooting section**](#troubleshooting_acquisition) + 🐞 If you don't see the log names supposed to be parsed, check the [**Acquisition Troubleshooting section**](#troubleshooting_acquisition)
- 🚨 If you don't see parsed lines, check the [**Collection Troubleshooting section****](#troubleshooting_collection) + 🐞 If you don't see parsed lines, check the [**Collection Troubleshooting section****](#troubleshooting_collection)
@@ -208,7 +212,7 @@ The CAPI allows your instance to receive a curated blocklist of malicious IP add ### Were all the tests successful ?
- 🚨 Connectivity Troubleshooting + 🐞 Connectivity Troubleshooting Check CAPI status with the command: sudo cscli capi status @@ -271,7 +275,7 @@ You might want to continue to the next recommended steps: - Then subscribe to more blocklists to benefit from additionnl proactive prevention
- 🚨 **Remediation Troubleshooting** + 🐞 **Remediation Troubleshooting** ... From d076d491f8aac7bc55574c3fc7b2b48f9292b4a4 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 23:12:13 +0200 Subject: [PATCH 40/46] improving detection section 2/2 --- .../post_installation/health_check.mdx | 79 +++++++++++-------- 1 file changed, 48 insertions(+), 31 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index 3d0e2c6d..145f70da 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -9,7 +9,7 @@ import CodeBlock from '@theme/CodeBlock'; import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; import HierarchicalList from '@site/src/components/HierarchicalList.js'; -Health Check Version: 0.1.0 +Health Check Version: 0.1.0 Welcome to the interactive Health-Check of your CrowdSec setup. We'll guide you through a series of tests to ensure that your Security Stack is fully functional and ready to protect your services: @@ -103,7 +103,7 @@ Were all the tests related to your setup successful? **This section will help you pinpoint the issue and walk you through how to fix it.**
- Are your logs being properly read and parsed? + 📄 Are your logs being properly read and parsed? CrowdSec needs to know what logs to read and how to interpret them. This is handled by the acquisition configuration (log sources) and parsing (how to read them). @@ -121,66 +121,83 @@ Were all the tests related to your setup successful? Under The **Parsers Metrics** you have the details of the parsers used. + 🚨 If this check fails, don’t worry -- the results will point you to the right area to troubleshoot: +
- 🐞 If this command fails, check the [**CrowdSec Service Troubleshooting section**](#troubleshooting_service) + 🐞 If this command fails entirely, go to the [**CrowdSec Service Troubleshooting section**](#troubleshooting_service)
- 🐞 If you don't see the log names supposed to be parsed, check the [**Acquisition Troubleshooting section**](#troubleshooting_acquisition) + 🐞 If your acquisition sources don't appear, check the [**Acquisition Troubleshooting section**](#troubleshooting_acquisition)
- 🐞 If you don't see parsed lines, check the [**Collection Troubleshooting section****](#troubleshooting_collection) + 🐞 If parsing fails, check the [**Collection Troubleshooting section****](#troubleshooting_collection)
- Acquisition Troubleshooting: are your logs properly declared as datasources - - Your acquisition configuration is located in the `crowdsec` directory under `acquis.yaml` or `acquis.d/anyName.yaml`. - The CrowdSec directory changes depending on your OS and installation method, for example, on Debian like OS it is usually located in `/etc/crowdsec/`. - - You may want to check your current acquisition configuration or [add a new acquisition file](https://doc.crowdsec.net/u/getting_started/post_installation/acquisition_new). - - Check that the datasource is properly setup: file path or correct port for syslog server... - - Ensure that the type declared matches the parser expected to be used: nginx, apache, syslog, etc. - - for more information, check the [documentation for the various datasources](https://doc.crowdsec.net/docs/next/log_processor/data_sources/intro) + 📥 Acquisition Troubleshooting -- Are your logs properly declared as datasources + + CrowdSec needs to know where to **read your logs**. This is handled by the **acquisition configuration**, usually found in `acquis.yaml` or in files under `acquis.d/` inside the CrowdSec config directory. + On Debian like OS it is typically located in `/etc/crowdsec/`. + + To troubleshoot: + - The detailed doc about the acquisition configuration can be found [here](https://doc.crowdsec.net/docs/next/log_processor/acquisition/intro). + - Check your acquisition files exist and that the datasources are properly setup. + - 💡 Hint: + - The hub page of the collection you installed provides an example of the acquisition configuration file to create. + - For example: + - The [NGINX collection hub page](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx) + - Or the [SSHD collection hub page](https://app.crowdsec.net/hub/author/crowdsecurity/collections/sshd) (that is contained in the Linux Collection). + - Make sure that the **type** declared in the matches the **parser** expected to be used: nginx, apache, syslog, etc.
- Collection Troubleshooting: are the parsers and scenarios installed? + 📦 Collection Troubleshooting -- Are the right parsers and scenarios installed? - #### Collection installation - - On host installations your existing service should have been detected and appropriate collections (containing parsers and scenarios) installed automatically. - - How ever on other kind of installations, or in case of failed detection, you may need to install the collections manually. + CrowdSec, via its [**Hub**](https://app.crowdsec.net/hub/collections) uses collections to package correct parsers and detection scenarios for your services. + - On regular **host** installations, CrowdSec usually detects your services (like nginx or ssh) and installs the appropriate collections automatically. + - On **Docker**, **Kubernetes**, or **custom setups**, you may need to install them manually. - Check the installed collections with the command: + #### 🔍 To check what's currently installed: sudo cscli collections list + You can also list individual parsers and scenarios with: sudo cscli parsers list sudo cscli scenarios list - - If you see the parsers and scenarios related to your service, then you're good to go. - - - To install new collection, check the [CrowdSec Hub](https://hub.crowdsec.net/) for a collection named after your service (e.g., [nginx](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx), [apache](https://app.crowdsec.net/hub/author/crowdsecurity/collections/apache), [linux](https://app.crowdsec.net/hub/author/crowdsecurity/collections/linux), etc.). - - If you find the collection, install it using the command shown on the hub page and follow the instructions provided for the acquisition file. - - #### Log format mismatch - - If your logs are in a custom format, you may need to create your own parsers or modify the existing ones to match your log format. + + - Look for entries related to your service (e.g., nginx, apache, ssh). + - If they’re listed, the right collection is likely installed. + + #### 📥 Install missing collections + 1. Visit the [CrowdSec Hub](https://hub.crowdsec.net/) and search for a collection matching your service, like: + - [nginx](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx) + - [apache](https://app.crowdsec.net/hub/author/crowdsecurity/collections/apache) + - [linux](https://app.crowdsec.net/hub/author/crowdsecurity/collections/linux) + - etc... + 2. Follow the installation instructions on the collection’s page, including any required acquisition setup. + + #### ⚠️ Log format mismatch + - If your logs don’t follow the expected format (e.g., they’ve been customized), CrowdSec might not parse them properly. - You can find more information on how to create your own parsers in the [CrowdSec documentation](https://doc.crowdsec.net/docs/next/log_processor/parsers/format).
- CrowdSec Service Troubleshooting: is the CrowdSec service running? - Make sure the CrowdSec service is running properly. You can check the status of the service with the following command: + ⚙️ CrowdSec Service Troubleshooting -- is the CrowdSec service running? + + Let’s check if the CrowdSec service is active: sudo systemctl status crowdsec - - You should see the service status as "active (running)". + - ☑️ You should see: "**active (running)**" - If the service is not running, you can start it with the following command: + If the service is not running, you can start it manually: sudo systemctl start crowdsec + - 💡 make sure it will be up after restart, activate the service If the service fails to start, you can check the logs for more information: sudo journalctl -u crowdsec - Reasons for the service not running can include: + Common reasons the service might fail:: - Misconfiguration in the `config.yaml` file. - Port conflicts with other services. By default, CrowdSec uses port 8080 for the Local API (LAPI) and port 6060 for the Local API metrics. - The port configuration can be setup in `config.yaml` file or by setting the environment variables depending on your implementation From fd34bd7983d2e7ff06225150059efc16cc3240b0 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 23:14:36 +0200 Subject: [PATCH 41/46] clean up to essential --- .../src/components/HierarchicalList.js | 71 ------- .../src/components/InteractiveCheckbox.js | 189 ------------------ .../post_installation/health_check.mdx | 2 - .../interactive_se_install/01_install.mdx | 154 -------------- .../02_parsers_scenarios copy.mdx | 76 ------- .../03_acquisition copy.mdx | 74 ------- .../interactive_se_install/04_remediation.mdx | 53 ----- 7 files changed, 619 deletions(-) delete mode 100644 crowdsec-docs/src/components/HierarchicalList.js delete mode 100644 crowdsec-docs/src/components/InteractiveCheckbox.js delete mode 100644 crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx delete mode 100644 crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx delete mode 100644 crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx delete mode 100644 crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx diff --git a/crowdsec-docs/src/components/HierarchicalList.js b/crowdsec-docs/src/components/HierarchicalList.js deleted file mode 100644 index fba6250a..00000000 --- a/crowdsec-docs/src/components/HierarchicalList.js +++ /dev/null @@ -1,71 +0,0 @@ -import React, { useState } from 'react'; - -const HierarchicalList = ({ data = [] }) => { - const [activeItem, setActiveItem] = useState(null); - - const handleItemClick = (itemId) => { - setActiveItem(activeItem === itemId ? null : itemId); - }; - - return ( -
- {/* Main list */} -
- {data.map((item, index) => ( - -
handleItemClick(item.id)} - > -
- {item.label} -
- {index < data.length - 1 && ( -
—
- )} - - ))} -
- - {/* Sub-elements */} - {activeItem && ( -
-
- {data - .find(item => item.id === activeItem) - ?.children?.map((child, index, array) => ( - -
-
- - {child.label} - -
- {index < array.length - 1 && ( -
—
- )} - - ))} -
-
- )} - - {/* Instructions */} -
-

How to use:

-

- Click on any main list item to toggle its sub-elements. The sub-elements will appear below - the main list, scaled down by 50% and displayed inline with connecting dashes. -

-
-
- ); -}; - -export default HierarchicalList; \ No newline at end of file diff --git a/crowdsec-docs/src/components/InteractiveCheckbox.js b/crowdsec-docs/src/components/InteractiveCheckbox.js deleted file mode 100644 index 2878092c..00000000 --- a/crowdsec-docs/src/components/InteractiveCheckbox.js +++ /dev/null @@ -1,189 +0,0 @@ -import React, { useState, useEffect, createContext, useContext } from 'react'; - -// Global context to manage checkbox states across all instances -const CheckboxContext = createContext(); - -// Provider component to wrap your MDX content -export const CheckboxProvider = ({ children }) => { - const [checkboxStates, setCheckboxStates] = useState({}); - const [manualChecks, setManualChecks] = useState({}); - - const updateCheckbox = (id, isManuallyChecked) => { - setManualChecks(prev => ({ - ...prev, - [id]: isManuallyChecked - })); - }; - - const isCheckboxChecked = (id, references = []) => { - // Manual check has priority - if (manualChecks[id]) { - return { checked: true, type: 'manual' }; - } - - // Check if all references are checked (either manually or through their references) - if (references.length > 0) { - const allReferencesChecked = references.every(refId => { - if (manualChecks[refId]) return true; - // Recursively check if reference has its own references - return checkboxStates[refId]?.checked; - }); - - if (allReferencesChecked) { - return { checked: true, type: 'reference' }; - } - } - - return { checked: false, type: 'none' }; - }; - - useEffect(() => { - // Update all checkbox states when manual checks change - const newStates = {}; - Object.keys(checkboxStates).forEach(id => { - const checkbox = checkboxStates[id]; - newStates[id] = { - ...checkbox, - ...isCheckboxChecked(id, checkbox.references) - }; - }); - setCheckboxStates(newStates); - }, [manualChecks]); - - const registerCheckbox = (id, references = []) => { - setCheckboxStates(prev => ({ - ...prev, - [id]: { - references, - ...isCheckboxChecked(id, references) - } - })); - }; - - return ( - - {children} - - ); -}; - -// The main checkbox component -export const InteractiveCheckbox = ({ - id, - references = [], - label = '', - className = '' -}) => { - const context = useContext(CheckboxContext); - - // Fallback if context is not available (useful for development/SSR) - if (!context) { - console.warn('InteractiveCheckbox must be used within a CheckboxProvider'); - return ( -
-
- {label && {label}} -
- ); - } - - const { - checkboxStates, - manualChecks, - updateCheckbox, - registerCheckbox, - isCheckboxChecked - } = context; - - useEffect(() => { - if (registerCheckbox) { - registerCheckbox(id, references); - } - }, [id, references, registerCheckbox]); - - const handleClick = () => { - if (updateCheckbox) { - updateCheckbox(id, !manualChecks[id]); - } - }; - - const checkboxState = isCheckboxChecked ? isCheckboxChecked(id, references) : { checked: false, type: 'none' }; - const isChecked = checkboxState.checked; - const checkType = checkboxState.type; - - // Determine the color based on check type - const getCheckmarkColor = () => { - if (checkType === 'manual') return '#22c55e'; // green - if (checkType === 'reference') return '#3b82f6'; // blue - return '#d1d5db'; // gray (unchecked) - }; - - const getBackgroundColor = () => { - if (isChecked) return getCheckmarkColor(); - return 'transparent'; - }; - - return ( -
-
- {isChecked && ( - - - - )} -
- {label && ( - - )} -
- ); -}; - -// Export both components as default for easier importing -export default InteractiveCheckbox; \ No newline at end of file diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index 145f70da..9c0e892e 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -6,8 +6,6 @@ title: CrowdSec Security Engine Setup Health-Check import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; -import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; -import HierarchicalList from '@site/src/components/HierarchicalList.js'; Health Check Version: 0.1.0 diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx deleted file mode 100644 index a824729c..00000000 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/01_install.mdx +++ /dev/null @@ -1,154 +0,0 @@ ---- -id: se_install_01 -title: Security Engine Installation -pagination_next: user_guides/interactive_se_install/se_install_02 ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import CodeBlock from '@theme/CodeBlock'; -import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; - -# Interactive Security Engine Installation Guide - -Welcome! This interactive guide will help you set up your CrowdSec Security Engine and validate each step to ensure proper operation. -We'll guide you through detecting and remediating malicious behavior in your services' logs, regardless of your chosen implementation (on host, Docker, Kubernetes) - -:::info -Note that to go further later you'll be able to enable Application Security and benefit from our ever growing collection of Virtual Patching rules. We'll mention the appropriate dependencies as we go along. -Lastly, note that we'll cover a setup that focuses on an autonomous security engine parsing the logs of any services from local or remote servers. -In appendix we'll cover the possibility to have distributed security engines on each servers, centralizing their alerts on a central security engine. -::: - -## Install CrowdSec - -The first step is to install the CrowdSec Security Engine somewhere you'll have access to the logs of services you want to protect. -You can choose to install it directly on the host, in a Docker container, or in a Kubernetes cluster. - -For a first experience, we recommend installing it on a host machine, as it will allow you to easily access the logs of your services and test the remediation capabilities. -But if you're comfortable with Docker it also is a great way to get started and run CrowdSec in a containerized environment. - -### Instructions - -/* We'll have to see how precise and embeded we want the various guides to be with this page - If we want full guides in here we'd better be able to embed files or pieces of files rather than copy paste - */ - - -

Here are the various hosts you can install CrowdSec on:

- // mention guide for linux, Windows, macOS, FreeBSD, pfSense, OPNSense with links to the existing docs - - - - // mention the current guide u/getting_started/installation/docker and this page having all the env variable https://hub.docker.com/r/crowdsecurity/crowdsec -

To install CrowdSec in a Docker container, you can follow the instructions in our Docker Installation Guide.

-

For more information on the available environment variables, you can refer to the Docker Hub page.

- - - // mention the current guide u/getting_started/installation/kubernetes and this page having all the env variable https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx -

To install CrowdSec in a Kubernetes cluster, you can follow the instructions in our Kubernetes Installation Guide.

-

For more information on the available environment variables, you can refer to the Artifact Hub page.

- - - -### Verification - -Let's check that CrowdSec is running and able to retrieve the community blocklist ! - -#### Enroll your Security Engine into CrowdSec Console -For advanced monitoring and trouble shooting the CrowdSec Console is a great tool to visualize your Security Engine's activity and alerts. - - - -- [link to doc] -- You'll see a confirmation pop up in the console for enrollment if not you might have conectivity issue to the central API [link to troubleshooting section] -- You'll see the last heartbeat and the status of your Security Engine in the console -- You'll be able to check various configurations for the upcoming steps of the installation -- You'll be warned when a new version of CrowdSec is available - -#### CrowdSec installation health - - - -```bash -systemctl status crowdsec -``` -- You should see the service status as "active (running)". -- The port configuration can be setup in config.yaml file or by setting the environment variables depending on your implementation - -> [ ] Check that your Local API (LAPI) is properly running: - -```bash -sudo cscli machines list -``` -- You should see a list of machines, including the one you just installed CrowdSec on. -- For an autonomous install the Security Engine is running and connecting to it's own LAPI -- You should see a recent last heartbeat and a checkmark in status. - -> [ ] Check the metrics can be queried (optionnal for cscli metrics and prometheus): - -```bash -sudo cscli metrics -``` -- CrowdSec serves the metrics via a prometheus endpoint, check that it's able to run properly -- You should see various tables, most still empty except the Local API Machines Metrics - -> [ ] Setup hub-update // ?@seb case we need this ? - -- ... - -#### CrowdSec connectivity health - -> [ ] Check your security engine can connect to the Central API - -```bash -sudo cscli capi status -``` -- You should see: "You can successfully interact with Central API (CAPI)" -- Optionally additional status - - Sharing signals is enabled //+link to doc where to turn this on/off ? - - Pulling community blocklist is enabled //+link to doc where to turn this on/off ? - - Pulling blocklists from the console is enabled //+link to doc where to turn this on/off ? - -### 🚨 Troubleshooting -
- There could be ports conflicts with other services - - The Local API is running on port 8080 by default, and the Metrics server is running on port 6060 by default. - If you have other services running on these ports, you can change the ports in the configuration file `/etc/crowdsec/config.yaml` or by setting the environment variables `CROWDSEC_API_PORT` and `CROWDSEC_METRICS_PORT` when running CrowdSec in a container. -
-
- No connectivity to Central API - - Check that you have access to internet at least api.crowdsec.net - Check that your online api credentials exist /etc/crowdsec/config/online_api_credentials.yaml (default path) - Eventually reset them with the following command sudo cscli capi register -
-
- Enrollment in Console not working - - Make sure you are looking in the proper organization in the console: the enrollment key is linked to your organization. - If you have multiple organizations, you can switch using the organization selector in the top left corner of the console. - If you already checked that there are no connectivity issues, you can try to re-enroll using the --overwrite flag, effectively forcing the engine to link to your organization. -
-
- Can't see the latest version of the package - - Update your repository and install again. - If latest version not available, Might depend on the plateform you're installing it on (be patient, it will come) -
\ No newline at end of file diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx deleted file mode 100644 index 4f6601a8..00000000 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/02_parsers_scenarios copy.mdx +++ /dev/null @@ -1,76 +0,0 @@ ---- -id: se_install_02 -title: Parsers and Scenarios Selection -pagination_prev: user_guides/interactive_se_install/se_install_01 -pagination_next: user_guides/interactive_se_install/se_install_03 ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import CodeBlock from '@theme/CodeBlock'; - -# Selection of Parsers and Scenarios - -blabla - -## Choosing what you need to protect your services - -To detect malicious behaviors targeting your services the (CrowdSec) Security Engine will need t be able to parse your services' logs and apply detection rules on them AKA **Scenarios**. -Our parsers and Scenarios are available on the [CrowdSec Hub](https://hub.crowdsec.net/), and are regrouped into **Collections** to make it more convenient to install as a whole. -Let's see what parsers and Scenarios you need to install to protect your services, and how to install them. - -### Instructions - -A typical use case would be to want to protect a web server, having an open ssh access. -You'd want parsers for your webserver logs (e.g. Nginx, Apache) and for your SSH logs, as well as the corresponding Scenarios to detect malicious behaviors. -In the case of NGINX being your reverse proxy you would want the NGINX parser AND all the scenarios related to protecting HTTP: -The [nginx collection](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx) would be one you want to install. -- It embeds the NGINX parser, as well as the Base HTTP scenarios collection and an additionnal NGINX specific scenario triggerring on nginx request limit exceeded. -- You can install it with the following command: -```bash -crowdsec collections install crowdsecurity/nginx -``` - -To protect your SSH accessed Linux server you would want the [linux collection](https://app.crowdsec.net/hub/author/crowdsecurity/collections/linux) which includes the Linux parser and a set of scenarios to detect malicious SSH behaviors. -- You can install it with the following command: -```bash -crowdsec collections install crowdsecurity/linux -``` - -One way you cna proceed for your own usecase is: -1. look for a logparser specific for your service in the [parser's section of the hub](https://app.crowdsec.net/hub/log-parsers) -2. Once you have found the parser you need, check if there is a collection with the same name. If so, chose the colleciton it will be pre-packed with reommanded scenarios. -3. You can browse other collections too or individual scenarios you might want to install. - - Note that you're also able to [create your own scenarios if needed](https://doc.crowdsec.net/docs/next/log_processor/parsers/intro). -4. Run the install you can find in the corresponding hub page -5. Keep the hub page in mind there are usefull instrucitons to the next part of this guide. - -### Verification - -Let's check the installation of the parsers and scenarios was successfull. -After installing them you should restart the CrowdSec service to ensure the new parsers and scenarios are loaded: -```bash -sudo systemctl restart crowdsec -``` - -#### List of installed parsers and scenarios - -> [ ] Check Collections, parsers and scenarios are installed -```bash -cscli collections list -cscli parsers list -cscli scenarios list -``` -- You'll be able to verify that the parsers and scenarios you installed are listed in the output of these commands. - -> [ ] Check scenarios are showing up in the console -When clicking on the "Scenarios" part of your [Security engine tile in the CrowdSec console](https://app.crowdsec.net/security-engines), you should see the scenarios you installed listed there. -The tile itself should show you the count of scenarios that are installed on your Security Engine. - -### 🚨 Troubleshooting - -
- Missing Scenarios - - @seb @laurence any reason why a scenario or colleciton would not install ? folder issue ? config ? -
\ No newline at end of file diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx deleted file mode 100644 index eebccbbe..00000000 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/03_acquisition copy.mdx +++ /dev/null @@ -1,74 +0,0 @@ ---- -id: se_install_03 -title: Parsers and Scenarios Selection -pagination_prev: user_guides/interactive_se_install/se_install_02 -pagination_next: user_guides/interactive_se_install/se_install_04 ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import CodeBlock from '@theme/CodeBlock'; - -# Acquisition - -Acquisition setup indicates how to retrieve the logs (file, syslog,...) and what parser to use for those logs. -The configuration for all acquisitions is located in /etc/crowdsec/acquis.yaml, you can choose to split and store them in /etc/crowdsec/acquis.d/\.yaml (both method can coexist) - -## Setting up acquisition datasources for detection - -TODO: explain how to infer the type give link to collection of prefilled with nicely named placeholders example for all datasource. give 2 examples for each: one minimal and one with all optional fields (+how to use wildcard and exclude for files for example) - can we make a gtp prompt ? - -### Instructions -... per datasource ? - -### Verification - -Let's check that parsing is working for all installed parsers and logs. - -#### Check acquisition: all necessary logs are read and parsed properly - -> [ ] Files are properly read and parsed -- Use your services normally: navigate on your website, log in via ssh ... -- Check the metrics -```bash -sudo cscli metrics -``` -Alternatively to select only the metrics we're interrested in here: -```bash -sudo cscli metrics show acquisition parsers -``` -- You should see names of the log files you configured in the acquisition section, and the number of lines parsed for each of them. - - The number of "Lines parsed" should be non-zero for each of the files you configured in the acquisition section. and for most cases should be equal to the number of "lines read". -- The parsers metrics show you what parsers were successfully used. Look for the name of the parsers you installed in the previous step - -#### Run on existing logs if you have some - -> [ ] Check alerts in past logs -- Most services open to the internet for more than a few days will have some malicious activity in their logs. -- You can run the Security Engine on existing logs to check if it detects any malicious activity. -- ... blabla we have DSN commands in datasources docs. -- Try... -- You will probably see alerts raised when doing this command: -```bash -sudo cscli alerts list -``` -- The alerts will also show up in the console within a few minutes if you enrolled your Security Engine. - -### 🚨 Troubleshooting -If you never see any alerts: However unlikely it's possible that you're not being hit by attacks but first lets check what could be wrong. - -
- IPs in your logs not properly X-forwarded for - - - Check that the logs you are reading have the real Source IP and not the one of one of your proxies or load balancers. - - Local/Private IPs are whitelisted by default, so log with non x-forwarded IPs wont trigger alerts. //more details here.. link ... - - If you have a reverse proxy, make sure to use the `X-Forwarded-For` header in your logs. - - If you have a load balancer, make sure to use the `X-Real-IP` header in your logs. -
-
- You are using custom log formats - - The parsers we provide are made to parse default log formats of the supported services. - If you are using custom log formats, you will need to create your own parsers or modify the existing ones to match your log format. - - You can find more information on how to create your own parsers in the [CrowdSec documentation](https://doc.crowdsec.net/docs/next/log_processor/parsers/intro). -
\ No newline at end of file diff --git a/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx b/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx deleted file mode 100644 index f3e20565..00000000 --- a/crowdsec-docs/unversioned/user_guides/interactive_se_install/04_remediation.mdx +++ /dev/null @@ -1,53 +0,0 @@ ---- -id: se_install_04 -title: Parsers and Scenarios Selection -pagination_prev: user_guides/interactive_se_install/se_install_03 ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; -import CodeBlock from '@theme/CodeBlock'; - -# Remediation - -Now that the detection is setup, lets remediate those bad IPs. You can apply a remediation in many places. -The job of a remediation component (aka RC or bouncer) is to keep the list of IPs up to date within the context you want to use: from the firewall of your host iptables/nftables to your edge appliances by exposing the list of IPs to it (mirror bouncer) through AWS WAF or Cloudflare... See full list of remediation components here[link] -This list of IPs will be comprised of decisions taken by the security engine (based on scenarios), IPs from blocklists your Security Engine is subscribed to (by default the community blocklist), and manual additions you may do via various methods. - -## Choosing and testing a remediation component - -// Explain various ways and advantages (push to be able to appsec and has metrics) - -### Instructions -... - -//Help connect to SE -A remediation component can get its information from the security engine via a URL+TOKEN (some support Mutual TLS authentication) -If installed on a host with a running security engine, they usually automatically pair (you can check with cscli bouncers list) to change the pairing, follow the steps described in the documentation - -### Verification - -Let's check ... - -#### Bouncer connectivity -> [ ] Check that the bouncer is connected to the Security Engine -Check that the bouncer is registered properly and active (in cscli and in the console) -```bash -cscli bouncers list -``` -- You should see the bouncer you just installed in the list, with a recent heartbeat and a valid tick mark - -> [ ] Check that the bouncing is effective -- Have them add a decision for the IP of our canary automate (or use a free blocklist we use for the guide with 10 most aggressive IPs and one IP of our test automate canary) -- Have them provide a URL that supposedly is protected -- Have the canary try to reach and return a response -- ??? How do we do this if we don't have a canary? without asking them to ban themselves -// find an alternative while we don't have a canary system @thib @seb @laurence - -### 🚨 Troubleshooting - -
- Docker communication with bouncer - - Docker bouncer: warning docker chain for example with firewall bouncer for docker user chain... ask for more precise phrasing @laurence -
\ No newline at end of file From c7c760ece795fb3b3b58a5ca96a26a90edbb5547 Mon Sep 17 00:00:00 2001 From: jdv Date: Wed, 18 Jun 2025 23:41:32 +0200 Subject: [PATCH 42/46] Connectivity section improve 1/1 --- .../post_installation/health_check.mdx | 38 ++++++++++++------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index 9c0e892e..8bb0445b 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -207,39 +207,51 @@ Were all the tests related to your setup successful? ## 🔌 CrowdSec Connectivity checks -### *List the community-blocked IPs* +### Is your Security Engine receiving community blocklists? -Let's check that your Security Engine can communicate with the CrowdSec Central API (CAPI). This connection is essential for participating in the community and protecting your setup with the collective intelligence of the network. -The CAPI allows your instance to receive a curated blocklist of malicious IP addresses identified and verified by the entire CrowdSec network. This proactive defense, known as the Community Blocklist, protects you from aggressors before they even target your services. It also allows your Security Engine to report the malicious IPs it detects, contributing to the strength of the community. +Let’s confirm that your Security Engine can communicate with the CrowdSec Central API (CAPI). +This connection allows you to: +- Receive **Community Blocklists** -- curated IPs flagged as malicious by the global CrowdSec network. +- Receive additional Blocklists of your choice among the ones available to you. +- Contribute back -- sharing detected Malicious IPs triggering installed scenarios.
🔌 CrowdSec Central API connectivity - The most direct way to verify connectivity is to see if your instance has already received decisions from the community blocklist. + The most direct way to verify connectivity is to see if your instance has already received decisions from the Community Blocklist. 1️⃣ List decisions coming from CAPI - sudo cscli decisions list --origin CAPI - 2️⃣ If you see decisions listed, it means your Security Engine is successfully connected to the CAPI and receiving decisions. + ☑️ If you see decisions, you're connected and receiving threat intel. + + **Notes:** + - On a fresh install, it might take a few minutes before any decisions appear.
### Were all the tests successful ? +Were all the tests related to your setup successful? +👍 If so, you can proceed to the next phase of the health check: [Remediation Check](#-remediation-checks) + +🛠️ If not, check the troubleshooting section below. +
🐞 Connectivity Troubleshooting - Check CAPI status with the command: + Let’s verify your CAPI connection step-by-step. + + Check CAPI status: sudo cscli capi status - - Should show: "You can successfully interact with Central API (CAPI)" - - If this test fail, try to enrol to the console to check what's going on. + - Should show: + - `INFO You can successfully interact with Central API (CAPI)` + - Along with information about the connectivity config file path and if your Security engine is enrolled in CrowdSec console. - Other common issues that can prevent your Security Engine from connecting to the CAPI include: - - Check that the online api credentials `online_api_credentials.yaml` exist in your crowdsec config directory + Common issues include: + - Missing `online_api_credentials.yaml` in your CrowdSec config directory - If they don't exist, you can create them by running the command: sudo cscli capi register - - Check that the `crowdsec` service is running and properly configured to connect to the CAPI. - - Firewall rules blocking outbound connections to the CrowdSec Central API. + - Firewall rules blocking outbound connections to the CrowdSec Central API (*api.crowdsec.net*) - DNS resolution issues. - Proxy server configuration. - Connectivity issues within Docker containers. From f895d53ab432dd68833ea213da67a6f56d7140aa Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 19 Jun 2025 00:13:59 +0200 Subject: [PATCH 43/46] Remediation section improve 1/2 --- .../post_installation/health_check.mdx | 37 ++++++++++--------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index 8bb0445b..bbc14127 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -251,7 +251,7 @@ Were all the tests related to your setup successful? - Missing `online_api_credentials.yaml` in your CrowdSec config directory - If they don't exist, you can create them by running the command: sudo cscli capi register - - Firewall rules blocking outbound connections to the CrowdSec Central API (*api.crowdsec.net*) + - Firewall rules blocking outbound connections to the CrowdSec Central API (api.crowdsec.net) - DNS resolution issues. - Proxy server configuration. - Connectivity issues within Docker containers. @@ -262,32 +262,33 @@ Were all the tests related to your setup successful? ### *Validate Blocks or Captchas* -Now that we've confirmed that the Security Engine can detect threats, let's verify that your bouncers are correctly applying decisions and blocking malicious IPs. +Now that detection and connectivity are working, let’s verify that your bouncers are correctly applying remediation on malicious IPs. -::info A note on future improvements -In the future, this health-check will include an option to trigger a remote test from the CrowdSec Console, which will safely probe your defenses from an external IP. -For now, we will perform a manual test by temporarily banning your own IP address. Please read the following warning carefully before proceeding. -::: +**Prerequisite:** +To apply remediation with CrowdSec, you’ll need a bouncer — available for firewalls, web servers (like Nginx), reverse proxies, CDNs, cloud WAFs, edge appliances, [and more](https://doc.crowdsec.net/u/bouncers/intro).
✋🏻 Bouncer Remediation test -This test involves manually creating a ban decision against your own public IP address for a very short period (1 minute). +This test involves manually creating a **decision** against a public IP of one of your devices for a very short period (1 minute). -::danger BE CAREFUL: Risk of Self-Lockout +:::danger +BE CAREFUL -- Risk of Self-Lockout This procedure will temporarily block your access to the services protected by your bouncer. -Make sure to properly set the TTL to a low expiration time (1 minute). OR do it from a device with a different public IP address than the client you're using to setup CrowdSec. -:: +Make sure to properly follow the instructions to set the TTL to a low expiration time (1 minute). +OR do it from a device with a different public IP address than the client you're using to setup CrowdSec. +::: -1️⃣ Find out you public address by using the following command: +1️⃣ Find your public IP: curl api.ipify.org or curl curl ipinfo.io/ip -2️⃣ Create a ban decision for your public IP address with a TTL of 1 minute: +2️⃣ Add a ban decision for your IP (valid for 1 minute): sudo cscli decisions add ban --ip \ --duration 1m --reason "CrowdSec remediation test" -3️⃣ Try to access your service (website, api ...) from the same public IP address. \> You should be blocked by the bouncer. +3️⃣ Try accessing your service (e.g. website, API). from the same public IP address. + ➡️ You should be blocked by the bouncer. returning a forbidden response (HTTP 403) or a captcha challenge. 4️⃣ Wait for 1 minute, then check the decisions list to see if the decision has been removed @@ -299,12 +300,12 @@ If you were successfully blocked, congratulations! Your remediation setup is wor You might want to continue to the next recommended steps: - Enroll your Security Engine to the [CrowdSec Console](/u/getting_started/post_installation/console/) -- Then subscribe to more blocklists to benefit from additionnl proactive prevention +- Then subscribe to more blocklists to benefit from additional proactive prevention
🐞 **Remediation Troubleshooting** - ... + Before diving into troubleshooting, remember that a bouncer is a separate component that connects to the Security Engine and regularly pulls decisions (like bans or captchas) to apply them at its level (firewall, web server, etc.). If remediation isn’t working, it’s often due to issues in this communication loop.
Bouncer Configuration Troubleshooting @@ -327,6 +328,8 @@ You might want to continue to the next recommended steps:
-## Your feedback is important! +## 💬 Your feedback is important! -If you have any suggestions or improvements for this health check, please let us know by opening an issue on our [GitHub repository](https://github.com/crowdsecurity/crowdsec-docs/issues/new) or on our [discord](https://discord.gg/wGN7ShmEE8) \ No newline at end of file +Help us improve this health check guide! +[📨 Open an issue on GitHub](https://github.com/crowdsecurity/crowdsec-docs/issues/new) or +🗣️ Join the conversation on [Discord](https://discord.gg/wGN7ShmEE8) \ No newline at end of file From 51f548916a7387dff4b1b6ceb33b7394e7c2b077 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 19 Jun 2025 09:04:10 +0200 Subject: [PATCH 44/46] mini clean --- crowdsec-docs/unversioned/getting_started/installation/linux.mdx | 1 - 1 file changed, 1 deletion(-) diff --git a/crowdsec-docs/unversioned/getting_started/installation/linux.mdx b/crowdsec-docs/unversioned/getting_started/installation/linux.mdx index 60161360..4ba9c5c8 100644 --- a/crowdsec-docs/unversioned/getting_started/installation/linux.mdx +++ b/crowdsec-docs/unversioned/getting_started/installation/linux.mdx @@ -8,7 +8,6 @@ pagination_next: getting_started/next_steps import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; -import { CheckboxProvider, InteractiveCheckbox } from '@site/src/components/InteractiveCheckbox.js'; # Installation Linux From 5d1d879ddc5aa050ab59af083fa32a345ee187b2 Mon Sep 17 00:00:00 2001 From: jdv Date: Thu, 19 Jun 2025 22:58:26 +0200 Subject: [PATCH 45/46] retour PR --- .../post_installation/health_check.mdx | 89 ++++++++++--------- 1 file changed, 46 insertions(+), 43 deletions(-) diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index bbc14127..16f03f6e 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -38,7 +38,7 @@ We'll trigger the dummy scenario `crowdsecurity/http-generic-test` by accessing curl -I https://\/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl 2️⃣ Confirm the alert has triggered for the scenario `crowdsecurity/http-generic-test` -sudo cscli alerts list -s crowdsecurity/http-generic-test +sudo cscli alerts list | grep crowdsecurity/http-generic-test **Notes:** - Requests from private IP addresses won't trigger alerts (private IPs are whitelisted by default). @@ -55,7 +55,7 @@ We'll trigger the dummy scenario `crowdsecurity/ssh-generic-test` by attempting ssh crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl@\ 2️⃣ Confirm the alert has triggered for the scenario `crowdsecurity/ssh-generic-test` -sudo cscli alerts list -s crowdsecurity/ssh-generic-test +sudo cscli alerts list | grep crowdsecurity/ssh-generic-test **Notes:** - This scenario can only be triggered again after a 5-minutes delay. @@ -67,15 +67,13 @@ We'll trigger the dummy scenario `crowdsecurity/ssh-generic-test` by attempting If you've enabled an AppSec-capable bouncer with CrowdSec WAF, you can trigger the `crowdsecurity/appsec-generic-test` dummy scenario. It would have triggered along with the HTTP detection test, but it is worth mentioning here as well. -Here is how to trigger the `crowdsecurity/appsec-generic-test` dummy scenario by calling a *probe path* on your web server. - We'll trigger the dummy scenario `crowdsecurity/appsec-generic-test` by accessing a **probe path** on your web server. 1️⃣ Access your service URL with this path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl` curl -I https://\/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl 2️⃣ Confirm the alert has triggered for the scenario `crowdsecurity/appsec-generic-test` -sudo cscli alerts list -s crowdsecurity/appsec-generic-test +sudo cscli alerts list | grep crowdsecurity/appsec-generic-test **Notes:** - This scenario can only be triggered again after a 1-minute delay. @@ -145,15 +143,15 @@ Were all the tests related to your setup successful? - 💡 Hint: - The hub page of the collection you installed provides an example of the acquisition configuration file to create. - For example: - - The [NGINX collection hub page](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx) - - Or the [SSHD collection hub page](https://app.crowdsec.net/hub/author/crowdsecurity/collections/sshd) (that is contained in the Linux Collection). + - The [NGINX collection hub page ↗️](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx) + - Or the [SSHD collection hub page ↗️](https://app.crowdsec.net/hub/author/crowdsecurity/collections/sshd) (that is contained in the Linux Collection). - Make sure that the **type** declared in the matches the **parser** expected to be used: nginx, apache, syslog, etc.
📦 Collection Troubleshooting -- Are the right parsers and scenarios installed? - CrowdSec, via its [**Hub**](https://app.crowdsec.net/hub/collections) uses collections to package correct parsers and detection scenarios for your services. + CrowdSec, via its [**Hub** ↗️](https://app.crowdsec.net/hub/collections) uses collections to package correct parsers and detection scenarios for your services. - On regular **host** installations, CrowdSec usually detects your services (like nginx or ssh) and installs the appropriate collections automatically. - On **Docker**, **Kubernetes**, or **custom setups**, you may need to install them manually. @@ -168,10 +166,10 @@ Were all the tests related to your setup successful? - If they’re listed, the right collection is likely installed. #### 📥 Install missing collections - 1. Visit the [CrowdSec Hub](https://hub.crowdsec.net/) and search for a collection matching your service, like: - - [nginx](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx) - - [apache](https://app.crowdsec.net/hub/author/crowdsecurity/collections/apache) - - [linux](https://app.crowdsec.net/hub/author/crowdsecurity/collections/linux) + 1. Visit the [CrowdSec Hub ↗️](https://hub.crowdsec.net/) and search for a collection matching your service, like: + - [nginx ↗️](https://app.crowdsec.net/hub/author/crowdsecurity/collections/nginx) + - [apache ↗️](https://app.crowdsec.net/hub/author/crowdsecurity/collections/apache) + - [linux ↗️](https://app.crowdsec.net/hub/author/crowdsecurity/collections/linux) - etc... 2. Follow the installation instructions on the collection’s page, including any required acquisition setup. @@ -193,7 +191,8 @@ Were all the tests related to your setup successful? - 💡 make sure it will be up after restart, activate the service If the service fails to start, you can check the logs for more information: - sudo journalctl -u crowdsec + For linux systems, the logs are typically located in `/var/log/crowdsec.log`. + less /var/log/crowdsec.log Common reasons the service might fail:: - Misconfiguration in the `config.yaml` file. @@ -227,6 +226,7 @@ This connection allows you to: **Notes:** - On a fresh install, it might take a few minutes before any decisions appear. + - Restarting the CrowdSec service will force it to perform a first pull.
### Were all the tests successful ? @@ -237,20 +237,21 @@ Were all the tests related to your setup successful? 🛠️ If not, check the troubleshooting section below.
- 🐞 Connectivity Troubleshooting + 🐞 Connectivity Troubleshooting Let’s verify your CAPI connection step-by-step. Check CAPI status: sudo cscli capi status - - Should show: - - `INFO You can successfully interact with Central API (CAPI)` - - Along with information about the connectivity config file path and if your Security engine is enrolled in CrowdSec console. - Common issues include: + **Should show:** + - `INFO You can successfully interact with Central API (CAPI)` + - Along with information about the connectivity config file path and if your Security engine is enrolled in CrowdSec console. + + **Common issues include:** - Missing `online_api_credentials.yaml` in your CrowdSec config directory - If they don't exist, you can create them by running the command: - sudo cscli capi register + sudo cscli capi register - Firewall rules blocking outbound connections to the CrowdSec Central API (api.crowdsec.net) - DNS resolution issues. - Proxy server configuration. @@ -281,13 +282,12 @@ OR do it from a device with a different public IP address than the client you're 1️⃣ Find your public IP: curl api.ipify.org -or -curl curl ipinfo.io/ip 2️⃣ Add a ban decision for your IP (valid for 1 minute): -sudo cscli decisions add ban --ip \ --duration 1m --reason "CrowdSec remediation test" +sudo cscli decisions add --ip \ --duration 1m --reason "CrowdSec remediation test" -3️⃣ Try accessing your service (e.g. website, API). from the same public IP address. +⏳ *Wait a few seconds to ensure the decision is processed by the bouncer.* +3️⃣ Try accessing your service (e.g. website, API). from the same public IP address. ➡️ You should be blocked by the bouncer. returning a forbidden response (HTTP 403) or a captcha challenge. 4️⃣ Wait for 1 minute, then check the decisions list to see if the decision has been removed @@ -305,31 +305,34 @@ You might want to continue to the next recommended steps:
🐞 **Remediation Troubleshooting** - Before diving into troubleshooting, remember that a bouncer is a separate component that connects to the Security Engine and regularly pulls decisions (like bans or captchas) to apply them at its level (firewall, web server, etc.). If remediation isn’t working, it’s often due to issues in this communication loop. - -
- Bouncer Configuration Troubleshooting - - * Verifying that the bouncer is registered and active. - * Checking bouncer logs for errors. - * Troubleshooting communication issues between the bouncer and the CrowdSec LAPI. - * Firewall configuration for the bouncer. - * Specific troubleshooting steps for common bouncers (firewall-bouncer, nginx-bouncer, etc.).) - -
+ Before diving into troubleshooting, remember that a remediation components (AKA **bouncer**) is a separate component that connects to the Security Engine and regularly pulls decisions (like bans or captchas) to apply them at its level (firewall, web server, etc.). If remediation isn’t working, it’s often due to issues in this communication loop. + You can find more information about bouncers in the [Bouncers documentation](https://doc.crowdsec.net/docs/next/bouncers/intro). + The full list of available bouncers is available on the [CrowdSec Hub ↗️](https://app.crowdsec.net/hub/remediation-components).
- Profile Troubleshooting - - * Verifying profile configuration. - * Troubleshooting errors related to profile application. - * Understanding the impact of profiles on detection and remediation.) - + Is your Bouncer Installed and Connected to your Security engine + + - Check bouncers linked to your Security Engine: + sudo cscli bouncers list + You should see: + - The bouncer name + - A tick in the valid column indicating that the bouncer is properly registered and connected to your Security Engine. + - a recent `Last API pull` datasources + + - If your bouncer is not valid or not pulling it might be an issue with the bouncer configuration authentication in its configuration file. + - If you don't see your bouncer listed, you should add it + - You can try to re-register your bouncer with the command: + sudo cscli bouncers add + - Copy the provided token and paste it in your bouncer configuration file. + - Then restart the bouncer service. + + - If your bouncer is on a different machine, ensure it can reach the Security Engine Local API. + - If you are using a bouncer in a container, ensure that the container can reach the Security Engine Local API.
## 💬 Your feedback is important! Help us improve this health check guide! -[📨 Open an issue on GitHub](https://github.com/crowdsecurity/crowdsec-docs/issues/new) or -🗣️ Join the conversation on [Discord](https://discord.gg/wGN7ShmEE8) \ No newline at end of file +[📨 Open an issue on GitHub ↗️](https://github.com/crowdsecurity/crowdsec-docs/issues/new) or +🗣️ Join the conversation on [Discord ↗️](https://discord.gg/wGN7ShmEE8) \ No newline at end of file From f6995207e5690fead43fbdc3b8d94f7609810163 Mon Sep 17 00:00:00 2001 From: jdv Date: Fri, 20 Jun 2025 08:55:54 +0200 Subject: [PATCH 46/46] nano change --- .../getting_started/post_installation/health_check.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx index 16f03f6e..690b7ce2 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/health_check.mdx @@ -266,7 +266,7 @@ Were all the tests related to your setup successful? Now that detection and connectivity are working, let’s verify that your bouncers are correctly applying remediation on malicious IPs. **Prerequisite:** -To apply remediation with CrowdSec, you’ll need a bouncer — available for firewalls, web servers (like Nginx), reverse proxies, CDNs, cloud WAFs, edge appliances, [and more](https://doc.crowdsec.net/u/bouncers/intro). +To apply remediation with CrowdSec, you’ll need a bouncer — available for firewalls, web servers, reverse proxies, CDNs, cloud WAFs, edge appliances, [and more](https://doc.crowdsec.net/u/bouncers/intro).
✋🏻 Bouncer Remediation test