Skip to content

NTgitdude23/blackweb

BranchesTags
 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

94 Commits
bwupdate
bwupdate
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
blackweb.tar.gz
blackweb.tar.gz
 
 
checksum.md5
checksum.md5
 
 

Repository files navigation

Blackweb

Blackweb is a project that collects and unifies public blacklists of domains (porn, downloads, drugs, malware, spyware, trackers, bots, social networks, warez, arms sales, etc.) to make them compatible with Squid

Blackweb es un proyecto que recopila y unifica listas negras públicas de dominios (porno, descargas, drogas, malware, spyware, trackers, bots, redes sociales, warez, venta de armas, etc) para hacerlas compatibles con Squid

DATA SHEET

lst Black Domains txt tar.gz Squid Tested
blackweb.txt 3.339.273 76.0 MB 15.0 MB v3.5.x

DEPENDENCIES

git subversion squid bash tar zip wget piconv curl python idn2 xargs awk notify-send

GIT CLONE

git clone --depth=1 https://github.com/maravento/blackweb.git

HOW TO USE

blackweb.txt is already optimized. Download it and unzip it in the path of your preference and activate Squid RULE / blackweb.txt ya viene optimizada. Descárguela y descomprimala en la ruta de su preferencia y active la REGLA de Squid

Download and Checksum

wget -q -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.tar.gz && cat blackweb.tar.gz* | tar xzf -
wget -q -N https://raw.githubusercontent.com/maravento/blackweb/master/checksum.md5
md5sum blackweb.txt | awk '{print $1}' && cat checksum.md5 | awk '{print $1}'

Squid-Cache Rule

Edit: / Edite:

/etc/squid/squid.conf

And add the following lines: / Y agregue las siguientes líneas:

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
acl blackweb dstdomain -i "/path_to_lst/blackweb.txt"
http_access deny blackweb

Squid-Cache Advanced Rules

Blackweb contains millions of domains, therefore it is recommended: / Blackweb contiene millones de dominios, por tanto se recomienda:

acl whitedomains dstdomain -i "/path_to_lst/whitedomains.txt"
acl blackdomains dstdomain -i "/path_to_lst/blackdomains.txt"
acl blackweb dstdomain -i "/path_to_lst/blackweb.txt"
http_access allow whitedomains
http_access deny blackdomains
http_access deny blackweb

UPDATE

⚠️ WARNING: BEFORE YOU CONTINUE!

Update and debugging of blackweb.txt can take and consume many hardware resources and bandwidth. It is not recommended to run it on production equipment / La actualización y depuración de blackweb.txt puede tardar y consumir muchos recursos de hardware y ancho de banda. No se recomienda ejecutarla en equipos en producción

Blackweb Update

The update process of blackweb.txt consists of several steps and is executed in sequence by the script bwupdate.sh / El proceso de actualización de blackweb.txt consta de varios pasos y es ejecutado en secuencia por el script bwupdate.sh

wget -q -N https://raw.githubusercontent.com/maravento/blackweb/master/bwupdate/bwupdate.sh && chmod +x bwupdate.sh && ./bwupdate.sh
Bandwidth Check (optional)

To guarantee update execution, before starting, script check bandwidth (with Speedtest). If it is > 1 Mbit/s, update continues; else, it shows warning messages and it is recommended to interrupt update / Para garantizar la ejecución de la actualización, antes de comenzar, el script verifica el acho de banda (con Speedtest). Si es > 1 Mbit/s, la actualización continúa; de lo contrario, muestra mensajes de advertencia y se recomienda interrumpir la actualización

Capture Public Blacklists

Capture domains from downloaded public blacklists (see SOURCES) and unifies them in a single file / Captura los dominios de las listas negras públicas descargadas (ver FUENTES) y las unifica en un solo archivo

Domain Debugging

Remove overlapping domains ('.sub.example.com' is a subdomain of '.example.com'), does homologation to Squid format and excludes false positives (google, hotmail, yahoo, etc.) with a whitelist (whiteurls.txt) / Elimina dominios superpuestos ('.sub.example.com' es un dominio de '.example.com'), hace la homologación al formato de Squid y excluye falsos positivos (google, hotmail, yahoo, etc.) con una lista blanca (whiteurls.txt)

com
.com
.domain.com
domain.com
0.0.0.0 domain.com
127.0.0.1 domain.com
::1 domain.com
domain.com.co
foo.bar.subdomain.domain.com
.subdomain.domain.com.co
www.domain.com
www.foo.bar.subdomain.domain.com
domain.co.uk
xxx.foo.bar.subdomain.domain.co.uk

outfile:

.domain.com
.domain.com.co
.domain.co.uk
TLD Validation

Remove domains with invalid TLDs (with a list of Public and Private Suffix TLDs: ccTLD, ccSLD, sTLD, uTLD, gSLD, gTLD, eTLD, etc., up to 4th level 4LDs) / Elimina dominios con TLD inválidos (con una lista de TLDs Public and Private Suffix: ccTLD, ccSLD, sTLD, uTLD, gSLD, gTLD, eTLD, etc., hasta 4to nivel 4LDs)

domain.exe
domain.com

outfile:

.domain.com
Debugging Punycode-IDN

Remove hostnames larger than 63 characters (RFC 1035) and other characters inadmissible by IDN and convert domains with international characters (not ASCII) and used for homologous attacks to Punycode/IDNA format / Elimina hostnames mayores a 63 caracteres (RFC 1035) y otros caracteres inadmisibles por IDN y convierte dominios con caracteres internacionales (no ASCII) y usados para ataques homográficos al formato Punycode/IDNA

президент.рф
mañana.com
bücher.com
café.fr
köln-düsseldorfer-rhein-main.de
mūsųlaikas.lt
sendesık.com

outfile:

xn--d1abbgf6aiiy.xn--p1ai
xn--maana-pta.com
xn--bcher-kva.com
xn--caf-dma.fr
xn--kln-dsseldorfer-rhein-main-cvc6o.de
xn--mslaikas-qzb5f.lt
xn--sendesk-wfb.com
DNS Loockup

Most of the SOURCES contain millions of invalid and nonexistent domains (see internet live stats). Then, each domain is verified via DNS and are excluded from the list (fault.txt). This process may take. By default it processes in parallel ≈ 6k to 12k domains x min, depending on the hardware and bandwidth / La mayoría de las FUENTES contienen millones de dominios inválidos e inexistentes (vea internet live stats). Entonces se verifica cada dominio vía DNS y se excluyen de la lista (fault.txt). Este proceso puede tardar. Por defecto procesa en paralelo ≈ 6k a 12k dominios x min, en dependencia del hardware y ancho de banda

HIT google.com
FAULT testfaultdomain.com
TLD Block

Add Black TLDs to block any domain that contains one. Edit blacktlds.txt and add or remove the TLDs you want to block / Agrega Black TLDs para bloquear cualquier dominio que contenga alguno. Edite blacktlds.txt y agrege o elimine los TLDs que quiera bloquear

.adult
.porn
.xxx
.domain.adult
.domain.porn
.subdomain.domain.xxx
.domain.com

outfile:

.adult
.porn
.xxx
.domain.com
Run Squid with Blackweb

Run Squid with Blackweb and any error sends it to SquidError.txt on your desktop / Corre Squid con Blackweb y cualquier error lo envía a SquidError.txt en su escritorio

Check execution (/var/log/syslog):
Blackweb: Done 06/05/2019 15:47:14
Important about Blackweb Update
  • The default path of blackweb is /etc/acl. You can change it for your preference / El path por default de blackweb es /etc/acl. Puede cambiarlo por el de su preferencia
  • bwupdate.sh includes lists of domains related to cloud/sync (Mega, Dropbox, Pcloud, iCloud, etc), remote support (Teamviewer, Anydesk, logmein, etc) and telemetry (May contain false positives). They are commented by default (unless their domains are in the SOURCES. To block or exclude them you must activate the corresponding line in the script (# JOIN LIST), although is not recommended to avoid conflicts or false positives / bwupdate.sh incluye listas de dominios relacionados con cloud/sync (Mega, Dropbox, Pcloud, iCloud, etc), soporte remoto (Teamviewer, Anydesk, logmein, etc) y telemetría (Puede contener falsos positivos). Están comentadas por defecto (excepto que sus dominios estén en las FUENTES). Para bloquearlas o excluirlas debe activar la línea correspondiente en el script (# JOIN LIST), aunque no se recomienda para evitar conflictos o falsos positivos

SOURCES

Blacklists
Active
Inactive
Whitelist (URLs/TLDs)
Active
Inactive
Work Lists
Internals
Optionals
Work Tools
Internals
Externals

BACKLINKS

CONTRIBUTIONS

We thank all those who have contributed to this project. Those interested can contribute, sending us links of new lists, to be included in this project / Agradecemos a todos aquellos que han contribuido a este proyecto. Los interesados pueden contribuir, enviándonos enlaces de nuevas listas, para ser incluidas en este proyecto

Special thanks to: Jhonatan Sneider

DONATE

BTC: 3M84UKpz8AwwPADiYGQjT9spPKCvbqm4Bc

LICENCES

GPL-3.0

CreativeCommons maravento.com is licensed under a Creative Commons Reconocimiento-CompartirIgual 4.0 Internacional License.

© 2019 Maravento Studio

DISCLAIMER

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

About

Domains Blacklist for Squid

www.maravento.com/

Resources

Readme

License

GPL-3.0 license
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages

  • Shell 90.0%
  • Python 10.0%