Clark Boylan c499b57e16 Add robots.txt to our list servers ...

We've noticed that our uwsgi queues are filling up and a lot of requests
are being made to robots.txt which ends up 500/503 erroring. Add a
robots.txt file which allows crawling of our lists and archives with a
delay value in hopes this will cause bots to cache results and not fill
up the queue with repetetive requests.

Change-Id: I660d8d43f6b2d96663212d93ec48e67d86e9e761

2024-04-23 08:51:50 -07:00

..

conftest.py

More yaml.safe_load() in testinfra/conftest.py

2021-10-13 21:58:41 +00:00

test_adns.py

dns: abstract names

2023-04-19 09:53:10 +10:00

test_base.py

logrotate: don't use filename to generate config file

2023-02-14 07:03:17 +11:00

test_borg_backups.py

system-config-run-borg-backup: rename hosts to distro

2022-08-11 13:32:49 +10:00

test_bridge.py

Farewell limestone

2023-02-13 23:54:59 +00:00

test_codesearch.py

hound: enable detect-ref

2022-02-25 17:27:35 +11:00

test_eavesdrop.py

ptgbot: rename site to ptg.opendev.org

2021-10-07 09:43:39 +11:00

test_etherpad.py

Upgrade Etherpad's MariaDB to 10.11

2024-03-04 15:12:22 -08:00

test_gerrit.py

Block access to Gitiles

2022-02-18 21:39:27 +00:00

test_gitea_lb.py

Test gitea haproxy maxconn limits

2024-01-02 18:12:00 +00:00

test_gitea.py

Increase gitea db connection limit

2024-02-05 10:40:03 -08:00

test_grafana.py

testinfra: refactor screenshot taking

2021-09-07 12:54:26 +10:00

test_graphite.py

graphite: fix xFilesFactor

2022-06-28 18:41:17 +10:00

test_keycloak.py

Clarify testinfra socket name for keycloak rdbms

2024-02-09 17:35:03 +00:00

test_letsencrypt.py

Update a few s/bridge01/bridge99 references

2022-11-11 15:05:39 +11:00

test_lists_opendev_org.py

Add robots.txt to our list servers

2024-04-23 08:51:50 -07:00

test_meetpad.py

Use our jitsi-meet image for meetpad

2020-03-27 09:43:41 -07:00

test_mirror-update.py

Cleanup opensuse mirroring configs entirely

2024-03-18 15:49:43 -07:00

test_mirror.py

More completely disable ansible galaxy proxy testing

2024-04-05 13:40:14 -07:00

test_nodepool.py

Add new opendev.org nodepool launchers

2021-03-16 15:21:58 -07:00

test_ns.py

dns: abstract names

2023-04-19 09:53:10 +10:00

test_paste.py

paste : move testing host to paste99, remove https hacks

2022-07-07 10:02:46 +10:00

test_refstack.py

refstack: Edit URL of public RefStackAPI

2021-03-10 14:09:20 +11:00

test_registry.py

Deploy insecure-ci-registry.opendev.org on jammy

2023-06-23 13:21:48 -07:00

test_static.py

Revert registry.zuul-ci.org

2023-11-14 16:05:28 -08:00

test_tracing.py

Add Jaeger tracing server

2022-09-15 19:21:33 -07:00

test_zookeeper.py

Add zk test to check myid is set in service

2023-10-17 09:31:16 -07:00

test_zuul_db.py

Add a standalone zuul db server

2024-04-04 12:25:23 -07:00

test_zuul_executor.py

Remove gearman from Zuul

2022-02-01 13:52:47 -08:00

test_zuul_lb.py

Add Zuul load balancer

2022-02-10 13:24:42 -08:00

test_zuul_merger.py

Remove gearman from Zuul

2022-02-01 13:52:47 -08:00

test_zuul_preview.py

Add zp02 zuul-preview server to replace zp01

2023-06-01 10:05:46 -07:00

test_zuul_scheduler.py

Remove gearman from Zuul

2022-02-01 13:52:47 -08:00

util.py

testinfra: Update selenium calls

2022-10-20 09:00:43 +11:00